Apache Airflow Connector

[mhrng]

User story

As a data scientist/data engineer/AI Tools customer/user, I would like to create pipelines, author, schedule and monitor workflows as Directed Acyclic Graphs (DAGs) of tasks on my Apache Airflow instance where some tasks will run on OVHcloud AI Training thanks to a specific operator.

Prerequisite

• Apache Airflow >= version 2.2

Acceptance criteria

• I can use Apache Airflow to create my own workflow of AI Training tasks
• Airflow will be able to interface with OVHcloud AI Training to execute some tasks while respecting the API interface for launching a job.
• I can visualize pipelines running in production, monitor progress and troubleshoot issues when needed on Apache Airflow.
• My pipelines are:
  • dynamic through configuration as code (Python),
  • lean and explicit, with the ability to parameterize my scripts,
• Apache Airflow will interface with OVHcloud AI Training thanks to an Operator like 'OVHCreateAITrainingJobOperator', that can be called in a dag.
• Ability to use through Airflow CLI, API
• OVHcloud AI Training operator is referenced inside Airflow official website and documentation (here : https://airflow.apache.org/docs/ )
• Official OVHcloud documentation about how to get started

Follow, vote and give your feedback

You can follow this task with the notification on the right tab. Ask us anything here in the comments below, and vote with emojis for most requested items ! 👍 to vote for this issue

Discuss on Discord

Feel free to discuss with us on https://discord.gg/ovhcloud

[AntoineRoll]

FYI, I'm starting to work on it here: https://github.com/AntoineRoll/airflow-provider-ovhcloud