search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.85k stars 14.25k forks source link

add yaml/omegaconf vizualization in UI in "run w/ config". #17912

Closed paantya closed 2 months ago

paantya commented 3 years ago

Description

Can you please tell me, is it possible in params to draw the output in the form of yaml/omegaconf, not json? in UI

  1. yaml is much more compact and easier to render.
  2. omegaconf - has convenient interpolation functions that would be great to use when working with configs - I use it, I really like it.

omegaconf:

image

Use case/motivation

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

potiuk commented 3 years ago

That would be a nice feature to allow yaml input - it should be entirely possible (and seems super easy).

What we need is just a Dict form, and this is both yaml and json compatible. We could even actually even parse the entry with yaml parser rather with json and this would work pretty much out-of-the-box @msumit - maybe you can include it - every "valid json" is automatically also "valid yaml" by definition, so it should be rather easy (and maybe there should be an option to choose the "default" values generated in the field as either json or yaml -it could be a toggle and could be stored in cookies. The json-schema defintion can easily validate yaml input as well (we are doing it all the time in Airflow).

When it comes to omegaconf, I think it's an overkill. We do not need to merge configurations - each DAG's configuration is separate from another, and they will never be "large and complex enough" to justify the interpolations, macros etc. I am not sure how omegaconf could be useful (though I do not know omegaconf - i got that impression after looking at the page, so maybe I do not get how it could be helpful. Maybe you can explain it @paantya and pass some example where you would find it useful )?

msumit commented 3 years ago

@potiuk yeah we can include the yaml support as well, but maybe in the next diff, cause the current diff is already become pretty big and hanging for a long time.

potiuk commented 3 years ago

absolutely

jscheffl commented 1 year ago

Hi @paantya This feature request is stale for a long time and I got notice from some housekeeping. We on our side also thought a lot about YAML but never made it to propose an extension. YAML and JSON both are very easy to be supported in Python and standard libs are available.

Nevertheless the UI for triggering has changed a lot since the issue was reported first, meanwhile AIP-50 (see https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-50+Trigger+DAG+UI+Extension+with+Flexible+User+Form+Concept) was implemented to generate a user friendly trigger UI. JSON dict is still possible via API and also in the UI. UI is controlled via Params (see https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html)

Do you still see the need/demand for such feature? Or shall we close the ticket as long as we don't find a contributor?

paantya commented 1 year ago

@jens-scheffler-bosch

wow, I haven't had time to get acquainted with the options page yet, thanks. I don’t understand yet, is this the default now or can I change the display to JSON?

In my many years of experience, editing a yaml config is much easier than editing json. especially when it comes to lists...

I still see the point of displaying it in the UI as json, as, for example, is done in CLearML, and already inside you can immediately translate it into json and do all the necessary checks for values. Although, of course, I am more attracted to working with OmegaConf , but since you have implemented a lot using json, then, as they say. up to the community :)

jscheffl commented 2 months ago

I still had this a long time in my personal backlog, but as nobody opted for this and other priorities catched by focus, going to close this. Please re-open if there is still demand after the trigger DAG run UI.

paantya commented 1 month ago

yes, its actual