exercism / gleam-test-runner

GNU Affero General Public License v3.0
3 stars 10 forks source link

Make tests use local version of exercism_test_runner. Closes #51 #52

Open kljensen opened 9 months ago

kljensen commented 9 months ago

Previously the tests had a hard-coded version of exercism_test_runner. This is not ideal because the CI tests are not actually testing the code in the repo---it is instead testing a previous version.

There are a few pieces to this changeset.

1) The exercism_test_runner path is made local and relative in all of the gleam.toml files in the tests directory.

2) The gleam_stdlib versions are updated in those gleam.toml files to be consistent with the version used by the local exercism_test_runner.

3) An extra mount point is added in the Docker command that runs the tests so that the tests have access to the local version of exercism_test_runner.

kljensen commented 9 months ago

There's a one things that bothers me about this PR and on which I'd appreciate feedback. I'm using

[dependencies]
gleam_stdlib = "~> 0.33 or ~> 1.0"

which I saw used elsewhere in this repo but seems weird to me.

kljensen commented 9 months ago

Maybe a silly question, but how is the exercism_test_runner package published?

I don't know. But there's a GitHub Actions workflow in this repo that looks like it deploys to dockerhub. I didn't notice anything deploying to hex.

ErikSchierboom commented 9 months ago

The deploy pushes the docker image, not the package. The reason I'm asking is because if the package is published somewhere else, it would be better to do use the package from hex and not locally, as that better simulates the runtime behavior.

kljensen commented 9 months ago

The deploy pushes the docker image, not the package. The reason I'm asking is because if the package is published somewhere else, it would be better to do use the package from hex and not locally, as that better simulates the runtime behavior.

That would be surprising to me; however, I must admit I don't know how exercism is wired together. My naive intuition is that one would test the runtime behavior in exercism/gleam. You can see there, for example, which versions of this package are used in the solutions and such.

ErikSchierboom commented 9 months ago

That would be surprising to me; however, I must admit I don't know how exercism is wired together

Well, I built the deploy workflow so 😄

The Dockerfile has:

# Download the used Gleam packages eagerly as the test runner will not have
# network access to do so. They are also pre-compiled for performance when
# compiling test projects.
RUN cd packages \
  && gleam deps download \
  && gleam build

The packages folder has a gleam.toml file that looks like this:

name = "packages"
version = "0.1.0"

[dependencies]
gleam_bitwise = "~> 1.2"
gleam_otp = "~> 0.7 or ~> 1.0"
gleam_stdlib = "~> 0.32 or ~> 1.0"
simplifile = "~> 1.0"

[dev-dependencies]
exercism_test_runner = "~> 1.4"

That seems to confirm what I asserted before.

I think the only question is: how does the test runner get published? Maybe this is some manual action? @lpil could you explain how this works?

lpil commented 9 months ago

Hello! Yes the test runner is published to Hex using the gleam publish command.

I believe the goal here is to use these tests to test any changes to the test runner itself, but we also need to make sure that they continue to be run with the published version, as this is the one that will actually be used on Exercism.