search

stateful / runme

DevOps Notebooks Built with Markdown
https://runme.dev
Apache License 2.0
1.17k stars 38 forks source link

Better command execution in Docker #631

Open adambabik opened 3 months ago

adambabik commented 3 months ago

The current implementation of command execution in Docker is naive and more of a proof of concept. Runme executes a container using a shell as the entry point and a command with its arguments as parameters. This works well for simple use cases, but it does not support many features: sessions, env collection, multiple executions in the same context, and sharing of arbitrary files between a host and a container.

This proposal provides a unified approach to executing commands in Docker and other remote targets. It requires the runme server to be running on the target, and the runme CLI only connects to it via gRPC. For Docker, there may be additional options such as volume mounting.

Proposal

Remote execution in Docker can be turned on by setting the runtime option in runme.yaml:

runtime:
  docker:
    enabled: true

This is aligned with how the current PoC implementation works.

When a command is executed using runme beta run command1 command2 the following will happen:

  1. If a Docker image does not exist, runme will build it. It’s owner’s of Dockerfile responsibility to ensure that a compatible runme binary is in the image.
  2. runme will start a Docker container with the runme server and expose a port. It will mount the project defined in runme.yaml in the read-only mode.
  3. runme will connect to the server via gRPC and send commands using the API.
  4. When all commands will be executed, the container will be shut down.
jlewi commented 3 months ago

If a Docker image does not exist, runme will build it Does this imply i) docker is running locally? ii) images are stored locally?

What if I build my images in the Cloud (e.g. Cloud Build) and store them in GCR?

When all commands will be executed, the container will be shut down.

Does this inhibit the case where container lifecycle needs to be managed outside of RunMe? Can RunMe treat docker as a special case of remote execution where it knows how to manage the lifecycle of the container and does it for you?

How would other remote targets be specified? In your proposal, it seems like in your runme.yaml you can only specify local execution via docker. If you want to support remote would you want to specify a gRPC address? Can the target executor only be specified at the document level or can different cells run in different executors?