Adding tags to DAG runs

[asaff1]

Description

Currently I see only option to tag DAGs. I don't see such an option for DAG runs. It will be quite useful for me, to add and filter runs by tags.

Use case/motivation

I'd like to filter runs based on tags. For example, I have different DAGs (predict, train, etc.) that I could tag with tags like 'model_name=mymodel', then I can see all runs that used that model. This also gives the user a way to group dagruns based on keys like experiment=name, such management exists in other dag runners (e.g kubeflow)

Related issues

No response

Are you willing to submit a PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Yep. Would be a great feature to have. Maybe someone will pick it up and implement (but FYI. The fastest way to get something like tha timplemented, is to contribute it, second fastest is to find someone who will contribute it - and for example pay them to do so - other than that someone will have to volunteer and implement it - and since this is an open-source project, it's on a discretion of whoever signs up and take that one.

[asaff1]

@potiuk I might be able to contribute it. I can submit a PR. Any guidelines for that?

As I see what needs to be done, is to add a new DB model DagRunTag with key, value fields (in models/dagrun.py), then adding a many to many relationship between DagRun and DagRunTag. Then I need to add CRUD operations to the API, and finally for the UI I need to add a filtering / search by tags. Is that a good direction?

[potiuk]

Yep. And it could be done in stages as well You could even add the list of things to do in the description of this issue with

• [ ] thing to do
• [ ] another thing to do

And check them one-by-one while doing it.

Also I think that one needs a say from @bbovenzi and @pierrejeambrun from the UI perspective, because if we implement something like that, it needs to someohow fit into navigation patterns in the new UI

(Note that this might be Airflow 3 - only change anyway)

[pierrejeambrun]

Interesting. I think we can make that work in the UI in terms of display and filtering in the DagRun table.

The thing I am wondering is how and when will tags be added to a specific dagrun ? For dags it is a dag authoring time, which makes sense, for dag_runs it is less clear to me. Will a user need to do that manually, or do the runs inherit the DAG tags by default, etc, etc

[asaff1]

@pierrejeambrun For my use case it is beneficial enough to set tags at creation time, or manually after creation (from the UI / API).

[bbovenzi]

First, before we add a new table. Let's see if we can use or improve existing fields to achieve this.

• Filter the dags for which you want to see dag runs. filter dags by tags then show me runs from only those dags
• I wonder if we can filter by dag_run.conf values.

[pierrejeambrun]

For my use case it is beneficial enough to set tags at creation time, or manually after creation (from the UI / API).

Set tags at creation time what do you mean by that ?

[asaff1]

@pierrejeambrun I mean, setting tags when manually triggering DAGs via the UI, or by the REST API. (So I mean, at the creation time of dag runs.)