DockerRunLauncher does not include volume mounts with runtime container instance

Summary

Docker Deployed Pipelines executed via Dagit Playground do not respect docker-compose mounted volumes.

When doing local development, it can be desirable to containerize all your dependencies into a docker container and then execute them via a local instance of dagit. ( also deployed via docker )

However, to make this development loop more seamless, one might want to volume mount your repo.py file, such that any updates made to the pipeline are live. Pipeline runs launched via the DockerRunLauncher do not inherit -volume declarations, and therefor, the code sent to the run is stale.

Reproduction

updated from docker_deploy from docker-compose.yml

` ... docker_example_pipelines ... networks:

docker_example_network

custom_service_pipelines: container_name: custom_service_pipelines image: custome_service:latest environment: DAGSTER_POSTGRES_USER: "postgres_user" DAGSTER_POSTGRES_PASSWORD: "postgres_password" DAGSTER_POSTGRES_DB: "postgres_db" DAGSTER_CURRENT_IMAGE: "custom_service:latest" volumes:
/path/to/pipeline/repo.py:/opt/dagster/app/repo.py networks:
docker_example_network `

However, when updating the /path/to/pipeline/repo.py on the host machine, say adding or renaming a solid my_solid --> new_solid_name, dagit, the UI, seems to find the new solid, but when the pipeline is run via the playground, it fails on execution, complaining about the missing solid:

dagster.check.CheckError: Invariant failed. Description: pipe has no solid named new_solid_name. File "/usr/local/lib/python3.8/dist-packages/dagster/grpc/impl.py", line 86, in core_execute_run yield from execute_run_iterator(recon_pipeline, pipeline_run, instance) File "/usr/local/lib/python3.8/dist-packages/dagster/core/execution/api.py", line 836, in __iter__ yield from self.execution_context_manager.prepare_context() File "/usr/local/lib/python3.8/dist-packages/dagster/utils/__init__.py", line 446, in generate_setup_events obj = next(self.generator) File "/usr/local/lib/python3.8/dist-packages/dagster/core/execution/context_creation_pipeline.py", line 350, in orchestration_context_event_generator context_creation_data = create_context_creation_data( File "/usr/local/lib/python3.8/dist-packages/dagster/core/execution/context_creation_pipeline.py", line 157, in create_context_creation_data resource_keys_to_init=get_required_resource_keys_to_init( File "/usr/local/lib/python3.8/dist-packages/dagster/core/execution/resources_init.py", line 325, in get_required_resource_keys_to_init hook_defs = pipeline_def.get_all_hooks_for_handle(step.solid_handle) File "/usr/local/lib/python3.8/dist-packages/dagster/core/definitions/pipeline.py", line 470, in get_all_hooks_for_handle solid = self.solid_named(name) File "/usr/local/lib/python3.8/dist-packages/dagster/core/definitions/graph.py", line 201, in solid_named check.invariant( File "/usr/local/lib/python3.8/dist-packages/dagster/check/__init__.py", line 167, in invariant raise CheckError(f"Invariant failed. Description: {desc}")

One possible work around is:

do no include the custom_service in the dagster docker-compose.yml
instead up a second docker-compose.yml up that:
- runs only the custom_service container
- with the proper network / configuration
- copies, rather than volume mounts the repo.py file each time

It would greatly lessen the development loop if the instantiated custom_service container could include the volume mounts during DockerRunLauncher execution

Dagit UI/UX Issue Screenshots

Additional Info about Your Environment

Message from the maintainers:

Impacted by this bug? Give it a 👍. We factor engagement into prioritization.

dagster-io / dagster