Integrate Snowflake Notebook Scheduling with Apache Airflow

jacobcbeaudin commented 3 months ago

Description

Enable scheduling and execution of Snowflake Notebooks from Apache Airflow. This feature would allow users to orchestrate and manage Snowflake Notebooks directly within Airflow, leveraging Airflow's scheduling, dependency management, and monitoring capabilities.

Use case/motivation

The recent launch of Snowflake Notebooks offers a powerful environment for data engineering, analytics, and machine learning workflows. By integrating Snowflake Notebooks with Airflow, users can automate their data workflows more efficiently. This integration will enable users to:

Schedule regular execution of Snowflake Notebooks.
Incorporate Snowflake Notebooks into larger, multi-step data pipelines.
Use Airflow's monitoring and alerting features to manage the execution of Snowflake Notebooks.
Reduce the need for manual intervention, thereby increasing productivity and ensuring consistency in data processes.

This feature will greatly benefit data teams who already use Snowflake and Airflow, providing a seamless and automated way to manage their end-to-end data workflows.

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

boring-cyborg[bot] commented 3 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

blcksrx commented 3 months ago

@Lee-W please assign it in my name

blcksrx commented 3 months ago

@jacobcbeaudin Just to be clear, does not airflow SnowflakeSqlApiOperator sufficient with execute notebook sql statement?

EXECUTE NOTEBOOK notebook_name();

jacobcbeaudin commented 3 months ago

@jacobcbeaudin Just to be clear, does not airflow SnowflakeSqlApiOperator sufficient with execute notebook sql statement?
EXECUTE NOTEBOOK notebook_name();

@blcksrx Thank you for the reference. The SnowflakeSqlApiOperator with the EXECUTE NOTEBOOK statement is indeed a sufficient current solution. However, I anticipate Snowflake will soon offer options to specify parameters for notebooks. At that point, the SQL commands may become less user-friendly. My feature request aims to enhance user experience similar to Databricks jobs Airflow operators, which provide a more streamlined and intuitive interface for managing complex workflows.

blcksrx commented 3 months ago

@jacobcbeaudin @Lee-W I understand, let's close this issue then, once Snowflake announced the official documentation then we'll continue

Lee-W commented 3 months ago

Sounds good to me 👍 @jacobcbeaudin I'll close it for now. But please feel free to reopen it

apache / airflow