Maelstrom is a fast Rust, Go, and Python test runner that runs every test in its own container. Tests are either run locally or distributed to a clustered job runner.
I want to be able to maelstrom-run --tty a job spec for a unit test that doesn't include a shell. I want to be able to poke around in the container and run the test manually to see the output and diagnose the error.
I want to do the above, but be able to safely run a test that uses PATH. I'm happy to have to be more explicit and type things like /.maelstrom/bin/ls instead of ls.
Acceptance Criteria
A user can run maelstrom-run --tty --shell or similar (maybe --busybox or --toybox?), with a job spec that doesn't contain a shell, and be connected to a shell in the container.
When run this way, besides the shell, we should provide some tools that may be necessary in the investigation: ls, cp, etc.
There is an option to modify the PATH environment variable to use whatever directory we put the common utilities in as the last item in the PATH.
Implementation Considerations
We may want to use toybox instead of busybox for licensing reasons.
Assuming we use toybox/busybox, we'll have to decide how to get the binary bits to the client:
One approach would be to publish our own multi-platform container image on a public repository. This image would presumably just have a single layer with everything we wanted put in the directory we wanted (e.g. /.maelstrom/bin). We'd have to figure out how we kept that image up-to-date. For this to work, we'd need the ability to specify a layer that's based off of an image but only uses some named layers from the image.
Another approach would be to use somebody else's container image. The problem with this is that the binaries would probably live in the wrong directory (/bin instead of /.maelstrom/bin). We'd need a way to specify a layer that is based off of an image's layer, but with things renamed.
Finally, we could just build the image into maelstrom-run. When necessary, we'd write it out to disk as a manifest or tar file, then specify that to the client.
We definitely need to put the shell and utilities in a non-standard directory so they don't conflict with other paths in the container. While we could allow specifying this on the command line, that seems like overkill. I think we could probably get away with /.maelstrom/bin, /.maelstrom-bin, or /.mbin/. Having a short path may be useful in situations where the user doesn't want to override PATH because they want to run the test case exactly how Maelstrom runs it, but they want to be able to use the utilities we provide.
Notes
We may find that certain commands don't work properly without /proc, /system, or whatever else mounted. We may want to have yet another flag to put these things in the container so those tools will work. I don't imagine that would be part of this initial issue, though we may open a separate issue later.
Use Cases
maelstrom-run --ttya job spec for a unit test that doesn't include a shell. I want to be able to poke around in the container and run the test manually to see the output and diagnose the error.PATH. I'm happy to have to be more explicit and type things like/.maelstrom/bin/lsinstead ofls.Acceptance Criteria
maelstrom-run --tty --shellor similar (maybe--busyboxor--toybox?), with a job spec that doesn't contain a shell, and be connected to a shell in the container.ls,cp, etc.Implementation Considerations
/.maelstrom/bin). We'd have to figure out how we kept that image up-to-date. For this to work, we'd need the ability to specify a layer that's based off of an image but only uses some named layers from the image./bininstead of/.maelstrom/bin). We'd need a way to specify a layer that is based off of an image's layer, but with things renamed.maelstrom-run. When necessary, we'd write it out to disk as a manifest or tar file, then specify that to the client./.maelstrom/bin,/.maelstrom-bin, or/.mbin/. Having a short path may be useful in situations where the user doesn't want to overridePATHbecause they want to run the test case exactly how Maelstrom runs it, but they want to be able to use the utilities we provide.Notes
/proc,/system, or whatever else mounted. We may want to have yet another flag to put these things in the container so those tools will work. I don't imagine that would be part of this initial issue, though we may open a separate issue later.Definition of Done
CHANGELOG.mdupdated.doc/book/headupdated.