search

exercism / automated-tests

An overview space for Automated Testing on Exercism
9 stars 7 forks source link

How do you choose which version to test against? #29

Open neenjaw opened 4 years ago

neenjaw commented 4 years ago

As requested by @ihid, i've created this issue as a followup to my question in exercism/elixir-test-runner

How are you all choosing which language version to run the solution against?

In the elixir track, the repo targets versions 1.6-1.9 with TravisCI, but likely running the solution against all four major versions in that range is likely not the intent of the project.

So how do you choose? Newest? Stable?

As I also said, this may be a bigger issue for the elixir track to consider that maybe rather than targeting four versions, it should just target one, then it reduces the complexity of testing.

I am also posting @ihid's response as requested and I think it's useful in having this discussion.

@iHiD, what is the normal for tracks in exercism?

I think choose a version and stick to it. But as we've not deployed our first test-runner yet, we've not locked that in. I think somewhere we need to expose this to the student. Could you open an issue at https://github.com/exercism/automated-tests asking this question pls, and linking to this comment?

Originally posted by @iHiD in https://github.com/exercism/elixir-test-runner/issues/3#issuecomment-547424991

ErikSchierboom commented 4 years ago

How are you all choosing which language version to run the solution against?

For C# and F#, I've chosen to run the same language version as the one that is specified by default in the exercises. Both of these languages have a project file that is downloaded along with each test suite file, and that project file specifies an exact version of the runtime is targets. The test runner subsequently tests using that version.

coriolinus commented 4 years ago

For Rust, we actually use two different versions, both brought in at the point of building the runner dockerfile. The runner itself is built using most recent stable. This should update every six weeks or so. The student's solution is built with the most recent nightly edition.

Some students will submit solutions requiring nightly Rust. I'd be OK with the runner failing in those cases, though; nightly is only required for new or advanced features which probably want a mentor's attention anyway. The real reason I use nightly at all is that emitting the test output as JSON, last time I checked (August), counted as a "new or advanced feature". It was worth using the nightly compiler to get JSON reports instead of attempting to parse text output intended for human consumption.

NobbZ commented 4 years ago

Current plan for Erlang is, to run against the version that was current ~3 months ago.

This roughly covers my speed with updating exercise CI and documentation for the recommended version.

Though, for erlang I do not expect things to break because the compiler is to new.

ricemery commented 4 years ago

In Scala the sbt file specifies the language version. As of now, all the exercises are set to the same Scala version.

bkhl commented 4 years ago

I think this is something each test runner and/or analysis tool should be able to handle based on the solution. The strategy per language might need to vary. For example:

Python

Rust

I think it might be possible to get warnings about use of unstable features from Clippy, so if that is run by the analysis tool first, that might help.

Elixir

(similar to how Erik says he's doing it for C# and F#)

Implementation

This will require the Docker image having all the versions you might want to run the tests with, which is probably not appropriate.

Maybe the analysis tool should run first, and one of the outputs for it should be the name of the docker image to use for the test run? This could then also return nothing if the analysis tool encounters a syntax error or something, in which case the test runner wouldn't need to run at all.

NobbZ commented 4 years ago

One can specify the version, though mix.exs usually just contains a semantic bound like ~> 1.5 which is basically >= 1.5.0 and < 2.0.0.

Usually it will work with newer versions without problems, unless errors_as_warnings has been activated, that is when things become dangerous in the elixir and erlang ecosystem.

coriolinus commented 4 years ago

this is something each test runner and/or analysis tool should be able to handle based on the solution

I'm not excited by the prospect of needing to download and setup a particular Rust version based on the student's tests. TBH I'm not thrilled to have needed to go to nightly at all. I'd much rather say "your program may work great on nightly, but for automatic test results, your code needs to work on stable".

iHiD commented 4 years ago

I'm not excited by the prospect of needing to download and setup a particular Rust version based on the student's tests.

The machine that runs the test doesn't have internet, so yeah, this is definitely not going to happen.

"your program may work great on nightly, but for automatic test results, your code needs to work on stable".

Agree.

iHiD commented 4 years ago

So a general question here, is how we tell students what version their code is going to run against. Where do we put that information so that the website can display it? Maybe in the config.json of the tracks? Thoughts?

coriolinus commented 4 years ago

how we tell students what version their code is going to run against

Wherever we put that information, the Dockerfile needs to have access at build time. No point duplicating information, after all.

The other option is to assume that the Dockerfile is the source of truth (as it is, de facto, anyway) and add a requirement that it include a file in a canonical location which reports the test runner's version.

SaschaMann commented 4 years ago

For the Julia track the plan is to use the latest stable x64 version. It uses semver versioning so it will be backwards compatible. The testsuites themselves are tested against all Julia versions with official support, so they're also backwards compatible. That way students can use the latest features if they like but submissions by students on older or LTS versions will also work. If they use release candidates or nightly builds, they should be used to things that don't work :D

how we tell students what version their code is going to run against

The other option is to assume that the Dockerfile is the source of truth (as it is, de facto, anyway) and add a requirement that it include a file in a canonical location which reports the test runner's version.

Couldn't the website extract that information from the Dockerfile directly?

iHiD commented 4 years ago

Couldn't the website extract that information from the Dockerfile directly?

I'm going to say no. I don't want to have to work out how to parse 50 Dockerfiles to get the relevant string, and I figure even if I did this would be very brittle. I think we'll need a file in a specific place, or an addition to an existing config file.

neenjaw commented 4 years ago

Okay, so I think what I'm gathering for the elixir track, I need to pose this question to the track maintainers and propose to have them decide on a version so that the test runner and the track can move in lock step.

@iHiD, i'm not sure what all the other languages would need to adequately specify their needs, but elixir should only need a version number, wherever that should be located.

iHiD commented 4 years ago

I need to pose this question to the track maintainers and propose to have them decide on a version so that the test runner and the track can move in lock step.

I think so, yes.

And then we need to collectively decide where to store that version so that I can easily get it to show to a user, which I increasingly feel would be best as a string in config.json.

coriolinus commented 4 years ago

The strongest argument against putting it in config.json is that it puts key information--the version--in two places: both the Dockerfile and the configuration. It will be much simpler to keep things in sync if the user-visible version information lives in a text file at a canonical path in the docker image, so that a single change updates things everywhere. Right now, config.json and Dockerfile for the test generators aren't even in the same repository. I've done my share of parallel PRs to keep distinct repos synced, and I'd prefer to avoid that in the future if possible!

On Wed, Oct 30, 2019 at 2:59 PM Jeremy Walker notifications@github.com wrote:

I need to pose this question to the track maintainers and propose to have them decide on a version so that the test runner and the track can move in lock step.

I think so, yes.

And then we need to collectively decide where to store that version so that I can easily get it to show to a user, which I increasingly feel would be best as a string in config.json.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/automated-tests/issues/29?email_source=notifications&email_token=AB3V4TUY2Z2MHPDAJXZIFZ3QRGHKXA5CNFSM4JGKDNW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECUI3WY#issuecomment-547917275, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3V4TWDZN3EAFZGP6HGRXTQRGHKXANCNFSM4JGKDNWQ .

iHiD commented 4 years ago

I've done my share of parallel PRs to keep distinct repos synced, and I'd prefer to avoid that in the future if possible!

I agree with your concerns about maintaining it in two places.

It will be much simpler to keep things in sync if the user-visible version information lives in a text file at a canonical path in the docker image.

The website doesn't interact with Docker images though, so we can't keep it in there.

I don't mind having a file at the base of each directory if we want to do that, but I think we're then still maintaining the same thing in two places. It's important that the Ruby track is tested on Travis against the same version of Ruby as the test-runner, and the analyzer and representer. All these things should be on the same version.

How about we have a github action that automatically applies any changes to a lang-version key in the track's config.json to a lang-version file in the analyzer, representer and test-runner repos, which the Dockerfiles can read from when they build? That keeps everyting in sync and removes any manual burden on maintainers.

coriolinus commented 4 years ago

I haven't played with github actions at all. If they can be set up to forward files between repos on changes like that, then yes, that's a good solution.

On Wed, Oct 30, 2019 at 3:16 PM Jeremy Walker notifications@github.com wrote:

I've done my share of parallel PRs to keep distinct repos synced, and I'd prefer to avoid that in the future if possible!

I agree with your concerns about maintaining it in two places.

It will be much simpler to keep things in sync if the user-visible version information lives in a text file at a canonical path in the docker image.

The website doesn't interact with Docker images though, so we can't keep it in there.

I don't mind having a file at the base of each directory if we want to do that, but I think we're then still maintaining the same thing in two places. It's important that the Ruby track is tested on Travis against the same version of Ruby as the test-runner, and the analyzer and representer. All these things should be on the same version.

How about we have a github action that automatically applies any changes to a lang-version key in the track's config.json to a lang-version file in the analyzer, representer and test-runner repos, which the Dockerfiles can read from when they build? That keeps everyting in sync and removes any manual burden on maintainers.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/automated-tests/issues/29?email_source=notifications&email_token=AB3V4TU6G47YP6H2KB7BGXDQRGJNJA5CNFSM4JGKDNW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECULBLA#issuecomment-547926188, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3V4TXMZVHX3TMFCHSEH2LQRGJNJANCNFSM4JGKDNWQ .

ricemery commented 4 years ago

Not to throw gas on the fire, but for the JVM languages a single version does not really tell the whole story. For the JVM languages - Scala, Clojure and Kotlin - there is the JVM version being used. And, there is the language version.

Lets take Scala as an example - we could be running the tests using JDK 1.8 and Scala language version 2.12.8. Currently the students know what to use by looking at the INSTALLATION.MD for the Scala track. And, each exercise specifies the Scala language version in the exercise build.sbt file. To make matters worse - each exercise could potentially use a different Scala version. Though they are all synced up now, and would want to keep the Scala versions in sync, it is a non trivial task when upgrading.

I am just not sure a single "version" is sufficient to identify the test system environment.

coriolinus commented 4 years ago

Couldn't Scala just use a version string like "JDK 1.8 / Scala 2.12.8"? That's pretty palatable for humans, but it's also easy to unpack into the individual versions necessary to construct an unambiguous Docker image. Then, it's just one more value to sync with the exercises.

On Wed, Oct 30, 2019 at 5:35 PM Ric Emery notifications@github.com wrote:

Not to throw gas on the fire, but for the JVM languages a single version does not really tell the whole story. For the JVM languages - Scala, Clojure and Kotlin - there is the JVM version being used. And, there is the language version.

Lets take Scala as an example - we could be running the tests using JDK 1.8 and Scala language version 2.12.8. Currently the students know what to use by looking at the INSTALLATION.MD for the Scala track. And, each exercise specifies the Scala language version in the exercise build.sbt file. To make matters worse - each exercise could potentially use a different Scala version. Though they are all synced up now, and would want to keep the Scala versions in sync, it is a non trivial task when upgrading.

I am just not sure a single "version" is sufficient to identify the test system environment.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/automated-tests/issues/29?email_source=notifications&email_token=AB3V4TU7ZUKNC6SISFPHE7TQRGZWVA5CNFSM4JGKDNW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECU4PQY#issuecomment-547997635, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3V4TXL3B4RW4MRIMZA4U3QRGZWVANCNFSM4JGKDNWQ .

ricemery commented 4 years ago

That would be fine as long as the value is just some human readable string. It is not clear to me how the value is expected to be used - and is expected to just be version number.

On Wed, Oct 30, 2019 at 9:42 AM Peter Goodspeed-Niklaus < notifications@github.com> wrote:

Couldn't Scala just use a version string like "JDK 1.8 / Scala 2.12.8"? That's pretty palatable for humans, but it's also easy to unpack into the individual versions necessary to construct an unambiguous Docker image. Then, it's just one more value to sync with the exercises.

On Wed, Oct 30, 2019 at 5:35 PM Ric Emery notifications@github.com wrote:

Not to throw gas on the fire, but for the JVM languages a single version does not really tell the whole story. For the JVM languages - Scala, Clojure and Kotlin - there is the JVM version being used. And, there is the language version.

Lets take Scala as an example - we could be running the tests using JDK 1.8 and Scala language version 2.12.8. Currently the students know what to use by looking at the INSTALLATION.MD for the Scala track. And, each exercise specifies the Scala language version in the exercise build.sbt file. To make matters worse - each exercise could potentially use a different Scala version. Though they are all synced up now, and would want to keep the Scala versions in sync, it is a non trivial task when upgrading.

I am just not sure a single "version" is sufficient to identify the test system environment.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/exercism/automated-tests/issues/29?email_source=notifications&email_token=AB3V4TU7ZUKNC6SISFPHE7TQRGZWVA5CNFSM4JGKDNW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECU4PQY#issuecomment-547997635 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AB3V4TXL3B4RW4MRIMZA4U3QRGZWVANCNFSM4JGKDNWQ

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/automated-tests/issues/29?email_source=notifications&email_token=AALYOG3G7FAQLFGJAD26TIDQRG2PRA5CNFSM4JGKDNW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECU5IHI#issuecomment-548000797, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALYOG4BB64CFFPRHEPSHMLQRG2PRANCNFSM4JGKDNWQ .

coriolinus commented 4 years ago

My impression is that it's a human-readable string, and it's the responsibility of the maintainer of the test runner / analyzer / whatever to parse that string within the Docker build and do the right things with it.

neenjaw commented 4 years ago

Even if it isn't, easy to do in structured way for ihid to capture for the purposes of the website:

{
  "language": "elixir",
  "version": "1.9.1",
  "copy": "Elixir 1.9.1, running on OTP 22",
  "env_versions": [
    {
      "name": "OTP",
      "version": "22"
    }
  ]
}
iHiD commented 4 years ago

With my website-maintainer hat on I care about a human-readable string we can display to students in the website. Track/tooling-maintainers will care about being able to reference that to avoid the syncing/duplication issues that @coriolinus has pointed out.

So I'm totally happy with track-maintainers parsing the human string, or having both a human, and a machine-readable version, encoded into the config.json.