search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.7k stars 1.48k forks source link

dagster-cloud CLI: run jobs in branch deployment by specifying branch name #24093

Open mmlakera opened 2 months ago

mmlakera commented 2 months ago

Dagster version

1.7.16

What's the issue?

My company is currently using Dagster Pro with hybrid deployment—and I wanted to figure out how I can use dagster-cloud job launch for branch deployments.

dagster-cloud job launch works for branch deployments if you specify the deployment id (a long hex string), but doesn't work when I attempt to specify the branch deployment name.

dagster-cloud job launch -l my_location -j my_job                  # works, triggers a job on the main deployment
dagster-cloud job launch -l my_location -j my_job -d db4a7dfa...   # works, triggers a job in the branch deployment
dagster-cloud job launch -l my_location -j my_job -d mm/my_branch  # ❌ does not work. See error below.

The last command throws the error

dagster_cloud_cli.core.errors.DagsterCloudAgentServerError: Exceeded 30 redirects.

Is this a bug? How can I launch jobs in a branch deployment by specifying the branch name?

Help would be appreciated!

What did you expect to happen?

dagster-cloud job launch -l my_location -j my_job -d mm/my_branch

launches a job in branch deployment mm/my_branch.

How to reproduce?

Launch a job on a branch deployment in Dagster+ Pro by specifying the branch name:

dagster-cloud job launch -l my_location -j my_job -d mm/my_branch

This will throw the following error:

...
dagster_cloud_cli.core.errors.DagsterCloudAgentServerError: Exceeded 30 redirects.

Deployment type

Dagster Pro, Hybrid

Deployment details

Workers are in a run in Kubernetes cluster in our AWS account

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

garethbrickman commented 2 months ago

Currently you have to use the ID for branch deployments because on the backend they don't have a deploymentName. This is also the case when submitting GraphQL queries to branch deployments.

You can see the ID from in the UI here: image