search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.5k stars 14.13k forks source link

Possibly add a feature to "Run Tasks" from the UI #31544

Open JonnyWaffles opened 1 year ago

JonnyWaffles commented 1 year ago

Apache Airflow version

2.6.0

What happened

Hi team, we just upgraded to 2.6.0 and now I cannot find the TaskInstance Context menu (detailed here in the docs). In the new UI, how do I force a task to run via web ui and ignore any dependencies?

What you think should happen instead

No response

How to reproduce

Open the UI click on a task instance, try to find the old Task Actions interface.

Operating System

ubuntu20.04

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Code of Conduct

Adaverse commented 1 year ago

Since you want to re run the task instance. You can clear the task instance and unselect the default selected options recursive and downstream (i believe) and clear. It should re run the task without affecting the deps.

PN - its not a bug. Its removed on purpose.

hussein-awala commented 1 year ago

As @Adaverse explained, it's still possible using the clear button, but the documentation is not up to date and needs to be updated.

Adaverse commented 1 year ago

Will update it with all the new UI screens

hussein-awala commented 1 year ago

@Adaverse assigned you

JonnyWaffles commented 1 year ago

Since you want to re run the task instance. You can clear the task instance and unselect the default selected options recursive and downstream (i believe) and clear. It should re run the task without affecting the deps.

PN - its not a bug. Its removed on purpos

Hi @Adaverse, I want to clear it and do a force rerun ignoring any dependencies and or skips. Is this action still available? It's not enough to just clear it since a down stream task is skipping. In the old UI I could choose ignore everything and run it similar to using the CLI flags -f -A -i.

hussein-awala commented 1 year ago

@JonnyWaffles Indeed, it appears that this is no longer possible, which seems to be a bug if I'm not mistaken.

@bbovenzi @pierrejeambrun any idea?

bbovenzi commented 1 year ago

The actions still exist for the graph and gantt views, we just dropped "Run":

Screenshot 2023-05-25 at 1 05 40 PM

Otherwise, in the grid view the actions are in the details nav:

Screenshot 2023-05-25 at 1 06 05 PM

We should update the docs to reflect that

JonnyWaffles commented 1 year ago

Right, so is it intended to remove the Run option entirely from the UI? Without Run I don't have a way to force trigger execution without using the CLI. Our environment is locked down so the CLI isn't an option. I need to use the UI to manually force a task.

pierrejeambrun commented 1 year ago

@JonnyWaffles Yes, it was removed in https://github.com/apache/airflow/pull/29706 because it was not working properly

JonnyWaffles commented 1 year ago

Okay so is there any other UI driven alternative to using the CLI to "force run" (-f -A -i) a task?

pierrejeambrun commented 1 year ago

I don't think so. I believe the idea was to use clear task to rerun a task, but this is not equivalent to the run options we had before.

We can convert this into a feature request, the goal would be to re-implement the run task from the UI but addressing issues stated in #29706. (maybe Brent has more context on how to do that)

harinipriya-nv commented 1 year ago

Hi All , We are also looking for the "run task" feature from UI. Can you please let me know if there is any update on this issue.

potiuk commented 1 year ago

Nope. The "run task" has been removed from the UI, also because it was a potential security issue, we will not get into details of that but a CVE announcement about it will be done soon (watch users@ and ASF announcements)

So the update here is - for now you can use clear task. Running single task from the UI is not likely to come back in its past form. I will turn it into feature request and I will encourage any one who wants somethign similar to explain their use case and why they cannot use any of the alternatives, including CLI commands (primary CLI commands were foreseen for such things like testing and running individual tasks).

Running single task on the UI and using webserver is not compliant with Airflow security model https://airflow.apache.org/docs/apache-airflow/stable/security/index.html, where users of the UI should not get the capabilities of running arbitrary code coming from the DAGs on the webserver.

AIrflow CLI on the other hand, with access to the Airflow servers is the place where running arbitrary code is expected and you have authorisation to do so. If it is just a "convenience" and "bypassing" of the security perimeter we set then it's not likely to come back as user feature and you should switch to CLI and running any kind of tests and tasks there if you need. If there is a legitmate feature, the please explain in detail the use case you have and why CLI access is not enough @JonnyWaffles @harinipriya-nv.

Hopefully if you do it in a clear an concise way and explain the use case, somebody will implement it. But this is like with all things in open source. If there is no-one doing it, it won't get implemented. Explaining the feature you want and reasoning why it is needed is the next best thing after proposing a PR doing it that you can do to increase the likelihood someone will pick an interest and implement it

Converting it into a feature request and waiting for the detailed explanation.

JonnyWaffles commented 1 year ago

Hi team, apologies for the delayed response @potiuk!

After reading through the security model, I don't really understand how force running a task is non-compliant, but then I'm no security engineer. Force running a task feels very similar in most use cases to clearing and letting the existing schedule rerun. We've always used it for administrative "break glass", where a single task may need to be executed regardless of its dependencies due to some external circumstances.

Our environment is extremely locked down, so the UI is the only real way to execute administrative tasks outside of a deployment pipeline. Even administrators cannot assume the Airflow identity required by CLI, because all production credentials are effectively machine read only. Only SSO, backed by multi-factor, login enables administration tasks. This feels appropriate to me because

I am definitely willing to work on this feature once it is prioritized by our internal team, but in the immediacy clearing is good enough, but I certainly miss the legacy feature.

pierrejeambrun commented 1 year ago

As explained above, the major concern was that the feature wasn't working as expected in all cases.

For the security part, there was a public advisory here: https://lists.apache.org/thread/j2nkjd0zqvtqk85s6ywpx3c35pvzyx15

Both of these problems are implementation related and not due to the nature of the feature itself, or conflicting with our security model.

With a correct design and implementation we can bring this back I believe.