search

ploomber / ploomber-engine

A toolbox 🧰 for Jupyter notebooks 📙: testing, experiment tracking, debugging, profiling, and more!
https://engine.ploomber.io
BSD 3-Clause "New" or "Revised" License
66 stars 14 forks source link

integration with Airflow #77

Closed edublancas closed 1 year ago

edublancas commented 1 year ago

hey @potiuk, moving the conversation from papermill to this repo so my team is aware of this. I'll start the discussion in the Airflow devlist shortly!

from @potiuk (source):

Ah. Nice!

Just thinking if maybe we should just a copy of existing papermill provider and do a "ploomber" one. I do see that papermill is pretty inactive so this would be a good idea.

Can you please start a discussion about this ("replace the papermill provider with ploomber-engine") at the devlist of Airlfow (see https://lists.apache.org/list.html?dev@airflow.apache.org) also https://airflow.apache.org/community/ has more information on joining the list. We have now pretty formal way of accepting new providers - especially if they are connected to existing services. So we would need to understand what is the relation between ploomber.io and ploomber-engine, can you use it with, or only without it and how much "open" the open-source (BSD licenced I see, which is cool) the engine is.

I think we need a bit more context (maybe links to some blogs and explanation why you decided to develop it) and I think we can take it from there.

potiuk commented 1 year ago

BTW. Papermill seems to be working - even if it has not been "officially" supporting Python 3.11 comment soon.

edublancas commented 1 year ago

the team doesn't have capacity to work on this, so I'm closing it