garethbrickman
commented
5 months ago In your dagster.yaml file, could you try adding an image key under config to specify which Docker image to use when launching runs. This image should be the one that contains your Dagster user code.
Example:
run_launcher:
module: dagster_docker
class: DockerRunLauncher
config:
image: "your_image_name:your_tag" # Replace with your actual image name and tag if it haso ne
env_vars:
- DAGSTER_POSTGRES_USER
- DAGSTER_POSTGRES_PASSWORD
- DAGSTER_POSTGRES_DB
network: dagster_network
container_kwargs:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /tmp/io_manager_storage:/tmp/io_manager_storage
seanpden
Dagster version
1.7.2
What's the issue?
I am attempting to deploy Dagster via a docker compose file. I've gotten the images all connected, the daemon and web server are both seeing the user code image, but when I attempt to materialize an asset, I get the following "ENGINE_EVENT" error:
The image name should be correct, below I have listed the output of
docker psI'm also adding my compose file, dagster.yaml file, and workspace.yaml file for reference.
An interesting note, I have propped up a working instance at work with an almost identical setup - the only obvious difference is the service names in my compose file. Most of this was ripped or tailored directly from the github repo listed in the docker deployment page.
compose.yaml
```sh services: dagster_metadata_db: image: postgres:13 container_name: dagster_metadata_db environment: - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} - POSTGRES_DB=${POSTGRES_DB} networks: - dagster_network dagster_user_code: build: context: . dockerfile: ./dockerfile_user_code container_name: dagster_user_code image: dagster-user-code environment: - DAGSTER_POSTGRES_USER=${POSTGRES_USER} - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD} - DAGSTER_POSTGRES_DB=${POSTGRES_DB} - DAGSTER_CURRENT_IMAGE="dagster-user-code" ports: - 4000:4000 networks: - dagster_network dagster_web_server: build: context: . dockerfile: dockerfile_web_and_daemon entrypoint: - dagster-webserver - -h - "0.0.0.0" - -p - "3000" - -w - "workspace.yaml" container_name: dagster_web_server image: dagster-web-server expose: - 3000 ports: - 3000:3000 depends_on: - dagster_user_code - dagster_metadata_db environment: - DAGSTER_POSTGRES_USER=${POSTGRES_USER} - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD} - DAGSTER_POSTGRES_DB=${POSTGRES_DB} - DAGSTER_CURRENT_IMAGE="dagster-web-server" volumes: - /var/run/docker.sock:/var/run/docker.sock - /tmp/io_manager_storage:/tmp/io_manager_storage networks: - dagster_network dagster_daemon: build: context: . dockerfile: dockerfile_web_and_daemon entrypoint: - dagster-daemon - run container_name: dagster_daemon image: dagster-daemon depends_on: - dagster_user_code - dagster_metadata_db networks: - dagster_network environment: - DAGSTER_POSTGRES_USER=${POSTGRES_USER} - DAGSTER_POSTGRES_PASSWORD=${POSTGRES_PASSWORD} - DAGSTER_POSTGRES_DB=${POSTGRES_DB} volumes: - /var/run/docker.sock:/var/run/docker.sock - /tmp/io_manager_storage:/tmp/io_manager_storage networks: dagster_network: driver: bridge name: dagster_network ```dagster.yaml
```sh scheduler: module: dagster.core.scheduler class: DagsterDaemonScheduler run_coordinator: module: dagster.core.run_coordinator class: QueuedRunCoordinator run_launcher: module: dagster_docker class: DockerRunLauncher config: env_vars: - DAGSTER_POSTGRES_USER - DAGSTER_POSTGRES_PASSWORD - DAGSTER_POSTGRES_DB network: dagster_network container_kwargs: volumes: # Make docker client accessible to any launched containers as well - /var/run/docker.sock:/var/run/docker.sock - /tmp/io_manager_storage:/tmp/io_manager_storage run_storage: module: dagster_postgres.run_storage class: PostgresRunStorage config: postgres_db: hostname: dagster_metadata_db username: env: DAGSTER_POSTGRES_USER password: env: DAGSTER_POSTGRES_PASSWORD db_name: env: DAGSTER_POSTGRES_DB port: 5432 schedule_storage: module: dagster_postgres.schedule_storage class: PostgresScheduleStorage config: postgres_db: hostname: dagster_metadata_db username: env: DAGSTER_POSTGRES_USER password: env: DAGSTER_POSTGRES_PASSWORD db_name: env: DAGSTER_POSTGRES_DB port: 5432 event_log_storage: module: dagster_postgres.event_log class: PostgresEventLogStorage config: postgres_db: hostname: dagster_metadata_db username: env: DAGSTER_POSTGRES_USER password: env: DAGSTER_POSTGRES_PASSWORD db_name: env: DAGSTER_POSTGRES_DB port: 5432 ```workspace.yaml
```sh load_from: - grpc_server: host: dagster_user_code port: 4000 location_name: "dagster-user-code" ```What did you expect to happen?
I expected the asset to materialize. The image name it is referencing seems to be correct and exists in according to the
docker pscommand.How to reproduce?
repo.py
```py from dagster import asset @asset def my_asset(): return ```dockerfile_user_code
```docker # syntax=docker/dockerfile:1 FROM python:3.10-slim RUN pip install \ dagster \ dagster-docker \ dagster-postgres \ dagster-webserver WORKDIR /opt/dagster/app COPY ./repo.py /opt/dagster/app/ EXPOSE 4000 CMD ["dagster", "api", "grpc", "--python-file", "repo.py", "--host", "0.0.0.0", "--port", "4000"] ```dockerfile_web_and_daemon
```docker FROM python:3.10-slim RUN pip install \ dagster \ dagster-webserver \ dagster-postgres \ dagster-docker ENV DAGSTER_HOME=/opt/dagster/dagster_home/ RUN mkdir -p $DAGSTER_HOME COPY dagster.yaml workspace.yaml $DAGSTER_HOME WORKDIR $DAGSTER_HOME ```docker compose up -d --buildDeployment type
Docker Compose
Deployment details
docker --version:OS: Ubuntu in WSL2
Additional information
Like I mentioned earlier, I had a build working that is very similar to this. Unsure of the difference other than a few image names.
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.