apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.62k stars 13.87k forks source link

DatabricksSubmitRunOperator libraries parameter has incorrect type #31877

Closed jlaneve closed 1 year ago

jlaneve commented 1 year ago

Apache Airflow version

2.6.1

What happened

The type of the libraries field in the DatabricksSubmitRunOperator is incorrect. According to the Databricks docs, the values should look more like:

[
    {"pypi": {"package": "simplejson"}},
    {"pypi": {"package": "Faker"}},
]

as opposed to what the type hint implies:

[
    {"pypi": "simplejson"},
    {"pypi":  "Faker"},
]

https://github.com/apache/airflow/blob/providers-databricks/4.2.0/airflow/providers/databricks/operators/databricks.py#L306

What you think should happen instead

No response

How to reproduce

n/a

Operating System

macOS

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

potiuk commented 1 year ago

cc: @alexott

phanikumv commented 1 year ago

I will check this