search

treeverse / airflow-provider-lakeFS

lakeFS airflow operator
Apache License 2.0
26 stars 5 forks source link

feat: update `with_metadata_operator.py` to support backwards compatibility with earlier versions than v2.5.0 #72

Closed FredrikBakken closed 1 year ago

FredrikBakken commented 1 year ago

Closes #71

arielshaqed commented 1 year ago

Yikes, provider test is failing 😭

If you can give the Airflow version(s) on which you did manage to run this code, it might help debugging.

FredrikBakken commented 1 year ago

I was able to run this on Apache Airflow version 2.4.3, but I did not try it again on any later versions. I'll try to see if I can make it run on the latest version (2.6.3) locally too, or if I encounter any problems. Reporting back to you soon!

FredrikBakken commented 1 year ago

I've now executed both the LakeFSCommitOperator and the LakeFSMergeOperator on a local Airflow instance, where both versions 2.4.3 and 2.6.3 have been able to run without any problems. However, I have not triggered a case where note is used and this field is therefore not showing in the LakeFS UI.

arielshaqed commented 1 year ago

Not sure why the provider test consistently fails. This is usually quite serious; I'll have time to look at it tomorrow.

FredrikBakken commented 1 year ago

I suspect I was able to pin-point the problem. I was using a dependency that does not exist in the current scope, package. I've changed it now to use the already installed packaging dependency. Hopefully this works better - and strange how it ran without any problems on my end... 🤔