search

datarootsio / terraform-aws-ecs-dagster

A terraform module that deploys Dagster to AWS, using ECS.
MIT License
30 stars 14 forks source link

Location syncing_job.py does not exist in workspace #7

Open CalRobert opened 2 years ago

CalRobert commented 2 years ago

This repo seems dead but I thought I'd share this anyway - I was never able to get this to work in ECS. Oddly, running in Docker locally with the same folder setup, workspace.yaml, etc was fine, but on ECS I always get this:

Caught an error for run b0e1a3fa-1dd4-4ee5-907c-1fe15cf59e0d while removing it from the queue. Marking the run as failed and dropping it from the queue: dagster._core.errors.DagsterRepositoryLocationNotFoundError: Location syncing_job.py does not exist in workspace

Anyone else have this problem?

adidonato commented 1 year ago

@CalRobert I got it working by adding the files to the app directory which I believe is what Dagster expects as default (at least from the docker image env vars). Same applies for new pipelines added to the yaml. The file needs to be available in app but the home dir where workspace.yml is ingested is a sibling dir, you need to mount the volume one directory up /opt/dagster/ and deal with the two sibling subdirs separately (if that makes sense).

My workspace.yaml :

load_from:
  - python_file:
      relative_path: /opt/dagster/app/syncing_pipeline.py
      location_name: /opt/dagster/app/
 - python_file: ....

I will soon push my working fork up so follow in case it doesn't get merged here.