Job groups - better way to organize jobs

[ruizh22]

What's the use case?

Today, for assets, there is the notion of asset groups, we can easily organize assets into groups and it is much easier to navigate on the UI with that hierarchy.

However, when it comes to jobs, there is no such hierarchy. We understand the number of jobs will be much less than the number of assets in most scenarios, but it would still be nice if we can group jobs so we can better organize them.

This feature will be especially helpful when we have jobs for different purposes (jobs for our main DAG and other operational jobs). And we don't have to suffer a UI sidebar with a long list of jobs in today's UI. Here is an example, and we have plans to add a lot more jobs in the future.

Ideas of implementation

Similar to how assets have groups, introduce groups/folders for jobs, and reflect the job grouping similar to asset grouping on the UI as well.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[sryza]

@braunjj - I know we've received similar requests in the past. Is there a central github issue where we're tracking these, or should this be the one?

[ruizh22]

hey @braunjj , any update on this one? It is not super urgent but it would be nice we can have it sooner.

[braunjj]

Hey @ruizh22. This is currently on our roadmap for this year, but we haven't started implementing it yet.

Our likely approach is to use definition-level tags to allow you to create groups of any definition type. So for example, you'd be able to add a tag named group:marketing to any number of Jobs, Assets, Sensors, Schedules or Resources, then view all your "marketing" definitions grouped in the UI. You'd be able to create nested groups as well by using a / in the tag value like group:marketing/source.

Along with this, we are exploring some changes to make these new definition-level groups easier to use and navigate in the UI. We're exploring the idea of creating a new "Definitions" tab in the main navigation where you can browse all your definitions and groups. This could potentially replace the left sidebar.

I recorded a demo of a concept we've been exploring here. I'd love to get your thoughts on this direction.

[bkozura]

Hi @braunjj, As it has been a few months since the last update, I wanted to check in on this and see if there was anything new on this. Has this feature been added to the roadmap?

Our dagster instance has 30+ pipelines and asset groups. For the meantime, we've used multiple create_repository_using_definitions_args() to create repository 'folders' to separate out related datasets and pipelines, but that also silos each repo from each other so we run into other workarounds if we ever need to reference other datasets. We are hoping to find another solution to organize all our jobs

[aq-ashwin]

Hey @braunjj, any update on this? The new UI would be very useful at 30+ jobs.

[WorksWellWithOthers]

Is this related to using Definitions and the jobs not being grouped?

I was using the older @repository and the jobs would be organized in the UI Overview but after switching over to Definitions I'm not sure how to group up similar jobs.

[kevorkkeheian]

This would be a great addition!

[LPauzies]

Just to follow the topic, i'm interested in it for a Dagster at scale.