Mazy Mice: Tests pass fine on the local machine, but fail on the server.

rabestro commented 10 months ago

Hi, @glennj , @IsaacG

Tests pass fine on the local machine, but fail on the server.

➜ mazy-mice git:(bash) ✗ bats test-mazy-mice.bats test-mazy-mice.bats ✓ the smallest square maze is perfect ✓ the small rectangular maze is perfect ✓ the square maze is perfect ✓ the large rectangular maze is perfect ✓ the rectangular maze with aspect 2:1 is perfect ✓ the huge rectangular maze is perfect ✓ the huge square maze is perfect ✓ if the seed parameter is specified, the perfect maze generated ✓ if the seed parameter is omitted, random mazes should be generated ✓ if the seed parameter is specified, the same maze should be generated

10 tests, 0 failures

➜ mazy-mice git:(bash) ✗ exercism submit mazy-mice.awk

Your solution has been submitted successfully.
View it at:
https://exercism.org/tracks/awk/exercises/mazy-mice

rabestro commented 10 months ago

only tests in which there is a call to this method fail

function validate_maze {
  result="$(\
    gawk --file mazy-mice.awk --assign Rows=$1 --assign Cols=$2 --assign Seed=$3 \
    | gawk --file test-maze.awk --assign Rows=$1 --assign Cols=$2 --assign Seed=$3 )"
  [ "$result" = "The maze is perfect." ]
}

glennj commented 10 months ago

Running it with the test runner on a linux VM:

$ pwd
/home/glennj/src/exercism/tooling/awk-test-runner
$ ls -l ~/src/exercism/tracks/awk/exercises/practice/mazy-mice/
.rw-r--r--  18k glennj 29 Aug 21:05 bats-extra.bash
.rw-r--r-- 1.9k glennj 29 Aug 21:05 mazy-mice.awk
.rw-r--r--  183 glennj 29 Aug 21:05 mazy-mice.awk.stub
.rw-r--r-- 1.8k glennj 31 Aug 19:48 results.json
.rw-r--r--  773 glennj 31 Aug 19:48 results.out
.rwxr-xr-x 3.4k glennj 29 Aug 21:05 test-maze.awk*
.rw-r--r-- 2.3k glennj 29 Aug 21:05 test-mazy-mice.bats

I've copied the example solution into mazy-mice.awk, keeping the stub solution file with the "stub" extension.

$ head -5 ~/src/exercism/tracks/awk/exercises/practice/mazy-mice/mazy-mice.awk
BEGIN {
    OFS = ""
    Rows = Rows ? Rows : 8
    Cols = Cols ? Cols : 16
    Seed ? srand(Seed) : srand()

And executing the tests locally

$ bin/run.sh mazy-mice $HOME/src/exercism/tracks/awk/exercises/practice/mazy-mice{,}
Running exercise tests for AWK
Test slug: mazy-mice
Solution directory: /home/glennj/src/exercism/tracks/awk/exercises/practice/mazy-mice
Output directory: /home/glennj/src/exercism/tracks/awk/exercises/practice/mazy-mice
Running tests.
Test output:
1..10
ok 1 the smallest square maze is perfect
ok 2 the small rectangular maze is perfect # skip
ok 3 the square maze is perfect # skip
ok 4 the large rectangular maze is perfect # skip
ok 5 the rectangular maze with aspect 2:1 is perfect # skip
ok 6 the huge rectangular maze is perfect # skip
ok 7 the huge square maze is perfect # skip
ok 8 if the seed parameter is specified, the perfect maze generated # skip
ok 9 if the seed parameter is omitted, random mazes should be generated # skip
ok 10 if the seed parameter is specified, the same maze should be generated # skip
Test run ended. Output saved in /home/glennj/src/exercism/tracks/awk/exercises/practice/mazy-mice/results.out
/home/glennj/src/exercism/tooling/awk-test-runner
Producing JSON report.
Tried to run 10 tests according to TAP plan.
Wrote report to /home/glennj/src/exercism/tracks/awk/exercises/practice/mazy-mice/results.json

and executing the tests in Docker shows the same errors

$ sudo bin/run-in-docker.sh mazy-mice /home/glennj/src/exercism/tracks/awk/exercises/practice/mazy-mice{,}
[sudo] password for glennj:
Sending build context to Docker daemon  56.32kB
Step 1/6 : FROM ubuntu:22.04
[...]
Step 2/6 : RUN apt-get update  [...]
Step 3/6 : WORKDIR /opt/test-runner
 ---> Running in cf8c1c518099
Removing intermediate container cf8c1c518099
 ---> f145eedaaf7a
Step 4/6 : COPY . .
 ---> 4d341fbee45d
Step 5/6 : ENV BATS_RUN_SKIPPED=true
 ---> Running in 27ca6321d3eb
Removing intermediate container 27ca6321d3eb
 ---> ade6dc7736f1
Step 6/6 : ENTRYPOINT ["/opt/test-runner/bin/run.sh"]
 ---> Running in 5b7febe96dea
Removing intermediate container 5b7febe96dea
 ---> 6a01e27c9e69
Successfully built 6a01e27c9e69
Successfully tagged exercism/awk-test-runner:latest
Running exercise tests for AWK
Test slug: mazy-mice
Solution directory: /solution
Output directory: /output
Running tests.
Test output:
1..10
not ok 1 the smallest square maze is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 12)
#   `validate_maze 5 5' failed
# Error: an invalid pattern of the top border
not ok 2 the small rectangular maze is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 17)
#   `validate_maze 5 10' failed
# Error: an invalid pattern of the top border
not ok 3 the square maze is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 22)
#   `validate_maze 10 10' failed
# Error: an invalid pattern of the top border
not ok 4 the large rectangular maze is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 27)
#   `validate_maze 10 20' failed
# Error: an invalid pattern of the top border
not ok 5 the rectangular maze with aspect 2:1 is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 32)
#   `validate_maze 20 10' failed
# Error: an invalid pattern of the top border
not ok 6 the huge rectangular maze is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 37)
#   `validate_maze 20 100' failed
# Error: an invalid pattern of the top border
not ok 7 the huge square maze is perfect
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 42)
#   `validate_maze 100 100' failed
# Error: an invalid pattern of the top border
not ok 8 if the seed parameter is specified, the perfect maze generated
# (from function `validate_maze' in file test-mazy-mice.bats, line 6,
#  in test file test-mazy-mice.bats, line 47)
#   `validate_maze 50 50 2342342' failed
# Error: an invalid pattern of the top border
ok 9 if the seed parameter is omitted, random mazes should be generated
ok 10 if the seed parameter is specified, the same maze should be generated
Test run ended. Output saved in /output/results.out
/opt/test-runner
Producing JSON report.
Tried to run 10 tests according to TAP plan.
Wrote report to /output/results.json

The results.json file is not more illuminating:

{
  "version": 3,
  "status": "fail",
  "test-environment": {
    "gawk": "GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)",
    "bats": "Bats 1.7.0",
    "OS": "Ubuntu 22.04.3 LTS"
  },
  "tests": [
    {
      "name": "the smallest square maze is perfect",
      "status": "fail",
      "test_code": "validate_maze 5 5",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 12)\n  `validate_maze 5 5' failed\nError: an invalid pattern of the top
border\n"
    },
    {
      "name": "the small rectangular maze is perfect",
      "status": "fail",
      "test_code": "validate_maze 5 10",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 17)\n  `validate_maze 5 10' failed\nError: an invalid pattern of the top
 border\n"
    },
    {
      "name": "the square maze is perfect",
      "status": "fail",
      "test_code": "validate_maze 10 10",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 22)\n  `validate_maze 10 10' failed\nError: an invalid pattern of the to
p border\n"
    },
    {
      "name": "the large rectangular maze is perfect",
      "status": "fail",
      "test_code": "validate_maze 10 20",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 27)\n  `validate_maze 10 20' failed\nError: an invalid pattern of the to
p border\n"
    },
    {
      "name": "the rectangular maze with aspect 2:1 is perfect",
      "status": "fail",
      "test_code": "validate_maze 20 10",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 32)\n  `validate_maze 20 10' failed\nError: an invalid pattern of the top border\n"
    },
    {
      "name": "the huge rectangular maze is perfect",
      "status": "fail",
      "test_code": "validate_maze 20 100",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 37)\n  `validate_maze 20 100' failed\nError: an invalid pattern of the top border\n"
    },
    {
      "name": "the huge square maze is perfect",
      "status": "fail",
      "test_code": "validate_maze 100 100",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 42)\n  `validate_maze 100 100' failed\nError: an invalid pattern of the top border\n"
    },
    {
      "name": "if the seed parameter is specified, the perfect maze generated",
      "status": "fail",
      "test_code": "validate_maze 50 50 2342342",
      "message": "(from function `validate_maze' in file test-mazy-mice.bats, line 6,\n in test file test-mazy-mice.bats, line 47)\n  `validate_maze 50 50 2342342' failed\nError: an invalid pattern of the top border\n"
    },
    {
      "name": "if the seed parameter is omitted, random mazes should be generated",
      "status": "pass",
      "test_code": "maze_one=$(gawk --file mazy-mice.awk --assign Rows=8 --assign Cols=16)\n# If the argument x is omitted, as in ‘srand()’, then the current date and time of day are used for a seed.\n# https://www.gnu.org/software/gawk/manual/html_node/Numeric-Functions.html#index-srand_0028_0029-function\n# So, we must wait a second to ensure the seed is different.\nsleep 1\nmaze_two=$(gawk --file mazy-mice.awk --assign Rows=8 --assign Cols=16)\n[[ $maze_one != \"$maze_two\" ]]"
    },
    {
      "name": "if the seed parameter is specified, the same maze should be generated",
      "status": "pass",
      "test_code": "maze_one=$(gawk --file mazy-mice.awk --assign Rows=8 --assign Cols=16 --assign Seed=123)\n# We must wait a second to ensure the system's default seed differs.\nsleep 1\nmaze_two=$(gawk --file mazy-mice.awk --assign Rows=8 --assign Cols=16 --assign Seed=123)\n[[ $maze_one == \"$maze_two\" ]]"
    }
  ]
}

glennj commented 10 months ago

I wonder if it's maybe a locale problem: I'd imagine the Ubuntu image doesn't have many locales installed, so it may be having trouble with the UTF chars.

glennj commented 10 months ago

I confirmed that changing the UTF maze characters passes the tests in Docker: using only - | + for the maze and > for the entry and exit (edits in the example solution and test-maze.awk)

$ sudo bin/run-in-docker.sh mazy-mice /home/glennj/src/exercism/tracks/awk/exercises/practice/mazy-mice{,}
Sending build context to Docker daemon  56.32kB
Step 1/6 : FROM ubuntu:22.04
 ---> 01f29b872827
Step 2/6 : RUN apt-get update                                   &&     apt-get install -y gawk jq git                   &&     git clone https://github.com/bats-core/bats-core &&     cd bats-core                                     &&     git checkout v1.7.0                              &&     bash ./install.sh /usr/local                     &&     cd ..                                            &&     rm -rf ./bats-core                               &&     apt-get remove -y git                            &&     apt-get purge --auto-remove -y                   &&     apt-get clean                                    &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 0224f7d36db4
Step 3/6 : WORKDIR /opt/test-runner
 ---> Using cache
 ---> f145eedaaf7a
Step 4/6 : COPY . .
 ---> Using cache
 ---> 4d341fbee45d
Step 5/6 : ENV BATS_RUN_SKIPPED=true
 ---> Using cache
 ---> ade6dc7736f1
Step 6/6 : ENTRYPOINT ["/opt/test-runner/bin/run.sh"]
 ---> Using cache
 ---> 6a01e27c9e69
Successfully built 6a01e27c9e69
Successfully tagged exercism/awk-test-runner:latest
Running exercise tests for AWK
Test slug: mazy-mice
Solution directory: /solution
Output directory: /output
Running tests.
Test output:
1..10
ok 1 the smallest square maze is perfect
ok 2 the small rectangular maze is perfect
ok 3 the square maze is perfect
ok 4 the large rectangular maze is perfect
ok 5 the rectangular maze with aspect 2:1 is perfect
ok 6 the huge rectangular maze is perfect
ok 7 the huge square maze is perfect
ok 8 if the seed parameter is specified, the perfect maze generated
ok 9 if the seed parameter is omitted, random mazes should be generated
ok 10 if the seed parameter is specified, the same maze should be generated
Test run ended. Output saved in /output/results.out
/opt/test-runner
Producing JSON report.
Tried to run 10 tests according to TAP plan.
Wrote report to /output/results.json

glennj commented 10 months ago

2 ways forward from here:

make the solution less pretty by not using UTF characters
install a UTF-8 local when building the image.

rabestro commented 10 months ago

Hi @glennj, thank you for looking into the issue! It appears that the bug only impacted the AWK and Bash tracks. It should not be an issue for Java or Python tracks.

I have another potential solution for the list.

Modify the test-maze.awk to adapt to the environment.

It should tolerate "broken" characters or adjust behavior based on the environment.

rabestro commented 10 months ago

I already have an ASCII-only option: https://github.com/rabestro/awk-maze-generator

➜  awk-maze-generator git:(release/5) ✗ gawk -f maze-gen-asc.awk -v Rows=9 -v Cols=21
######################################################################################
##          ##              ##          ##                          ##              ##
##  ######  ##  ##########  ######  ##  ##  ##############  ######  ##  ##  ######  ##
##  ##          ##      ##      ##  ##  ##      ##      ##      ##  ##  ##      ##  ##
##  ##  ##########  ##  ######  ######  ######  ##  ##########  ######  ######  ##  ##
##  ##  ##          ##  ##  ##      ##      ##  ##          ##          ##      ##  ##
##  ######  ##########  ##  ######  ##  ##  ##  ######  ##  ##############  ######  ##
##      ##  ##  ##      ##      ##  ##  ##  ##      ##  ##              ##      ##  ##
##  ##  ##  ##  ##  ######  ##  ##  ##  ##  ######  ##  ##############  ######  ##  ##
##  ##          ##  ##  ##  ##  ##  ##  ##      ##  ##          ##      ##      ##  ##
##  ##############  ##  ##  ##  ##  ##  ##########  ##############  ##  ##  ######  ##
##  ##              ##      ##          ##      ##      ##          ##  ##  ##  ##  ##
##  ##  ##  ##########  ##################  ##  ######  ##  ######  ######  ##  ##  ##
##  ##  ##  ##      ##  ##                  ##  ##      ##  ##      ##      ##        
##  ##  ######  ##  ######  ##################  ##  ######  ######  ##  ##############
    ##      ##  ##          ##      ##          ##      ##      ##  ##  ##          ##
##  ######  ##  ##############  ##  ##  ##############  ######  ##  ##  ##  ######  ##
##      ##                      ##  ##                          ##  ##          ##  ##
######################################################################################
➜  awk-maze-generator git:(release/5) ✗

rabestro commented 10 months ago

One more possible solution:

Preprocess the student's maze and replace UTF-8 with ASCII by tr or similar tools.

rabestro commented 10 months ago

https://github.com/exercism/problem-specifications/pull/2312

Vincent-de-Comarmond commented 3 weeks ago

Hello.

I have a similar, but slightly different issue. Like @rabestro my tests pass locally, but fail on the server .

The typical error (on the server) is something like this: `validate_maze 5 5' failed Error: expected 11 columns, got 33 at

With roughly 3x the expected number of columns being found.

I've yet to understand if this is an encoding problem and will try to look into it.

Vincent-de-Comarmond commented 3 weeks ago

Sample solution from exercises/practice/mazy-mice/.meta/example.awk also fails on the server.

rabestro commented 3 weeks ago

Sample solution from exercises/practice/mazy-mice/.meta/example.awk also fails on the server.

Hi @Vincent-de-Comarmond The issue is on the server side. We need to update the docker image to add support for Unicode, or I should change the task description, all tests, and the sample solution to use only ASCII graphics.

Unfortunately, it is not possible for now to pass the problem...

rabestro commented 3 weeks ago

@Vincent-de-Comarmond the solution with ASCII graphics looks like this: https://github.com/rabestro/awk-maze-generator

Vincent-de-Comarmond commented 3 weeks ago

@rabestro

Is there anything I can do to help with this?

exercism / awk

Mazy Mice: Tests pass fine on the local machine, but fail on the server. #213