Dagster daemon on GCR

[shane-breeze]

Thanks for putting together this terraform module to showcase deploying dagster on cloud run. I'm trying to adopt this to my use case and would prefer not to use cloud engine for the dagster daemon. Do you know if there's an easy way to package the dagster daemon into cloud run?

I've already tried this, but since the dagster daemon is not a HTTP/TCP/GRPC service GCR fails on the startup probe.

[timchap]

IMO this kind of daemon-style service is kind of explicitly outside the scope of Cloud Run. Cloud Run is built primarily for:

• Stateless services that have no processing outside of the request window (Cloud Run services)
• Finite jobs that have a well-defined termination state (Cloud Run jobs)

Daemons don't really fit into these two cases, which is why there isn't an easy solution. If you really wanted to, there are two solutions that might work:

• Create a service with min_replicas=max_replicas=1, and enable idle CPU. Create a container which runs the - Dagster daemon as a background process, and a simple service to respond to Cloud Run's health check.
• Create a Cloud Run job with the maximum timeout. Configure this job to automatically trigger itself as it approaches timeout.

For me, neither of these are particularly satisfying, so unless you really really want to avoid Compute Engine then my recommendation would be to provision the smallest possible instance.