astronomer / astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer
https://astronomer-providers.rtfd.io/
Apache License 2.0
135 stars 25 forks source link

Add Firebolt Provider #322

Open denimalpaca opened 2 years ago

denimalpaca commented 2 years ago

Is your feature request related to a problem? Please describe. The Firebolt team is interested in a deeper partnership with Astronomer, including the creation of astronomer-provider operators.

Describe the solution you'd like Work with the Firebolt team to develop operators that would fall under the astronomer-providers library.

Describe alternatives you've considered

Additional context

phanikumv commented 2 years ago

Hi @denimalpaca - just wanted to know which operators can we plan initially, do we have a Firebolt instance set up somewhere which the provider’s team can access?

phanikumv commented 2 years ago

For any async operator to be developed, we need a query id (something like a pid which is unique for each query), because we need to be able to poll for the query status on the Triggerer

denimalpaca commented 2 years ago

Closed by accident -- please ignore. No update on this right now.

kevinmarr commented 2 years ago

Note that the MVP provider has been built, which should close this feature request: https://registry.astronomer.io/providers/firebolt

@phanikumv , regarding your question about async, we are currently working on adding support for it in our Python driver, which will then make it possible to build an async operator.

kevinmarr commented 2 years ago

@phanikumv is your idea that the async operator would simply run the query async, return the query ID, and be done? Then the polling for the query status would happen via the Airflow scheduler?

phanikumv commented 2 years ago

yes @kevinmarr - the async operator would submit the query and defer to the Triggerer, where we would poll for the query status.

phanikumv commented 1 year ago

@kevinmarr any update on this ? Do you have an async version of your Python driver