search

zenml-io / mlstacks

A series of Terraform based recipes to provision popular MLOps stacks on the cloud.
https://mlstacks.zenml.io/
Apache License 2.0
245 stars 32 forks source link

Add Azure Modular Recipe/Stack Deployment Option #139

Open strickvl opened 5 months ago

strickvl commented 5 months ago

While MLStacks currently supports GCP and AWS for stack deployments, there is a notable absence of support for Azure. Although the creation of remote state on Azure is facilitated through a Terraform registry module implemented by ZenML), comprehensive support for Azure stack deployments through modular Terraform recipes is missing. This task aims to bridge that gap by introducing Azure as a fully supported provider within MLStacks.

Task Description

The objective is to develop a series of Terraform modules for Azure, analogous to those available for AWS and GCP within MLStacks. This will enable users to deploy selective parts of their MLStacks on Azure. The task involves leveraging existing Terraform code (e.g., azure-minimal and azureml-minimal) and adapting it to the modular approach used in MLStacks. Full testing is required to ensure the Azure implementation's functionality and compatibility with MLStacks.

Expected Outcome

Steps to Implement

  1. Review existing Azure-related Terraform code within MLStacks and related repositories to identify reusable components.
  2. Develop modular Terraform recipes for Azure, ensuring they align with the structure and approach used for AWS and GCP within MLStacks.
  3. Implement Azure stack components within MLStacks, ensuring they support the dynamic selection and deployment of various stack parts.
  4. Conduct thorough testing of the Azure deployment option across different configurations and scenarios to ensure full functionality and integration within MLStacks.
  5. Update the MLStacks documentation to include comprehensive guidance on using the Azure deployment option, covering the setup, configuration, deployment, and best practices.
  6. Engage with the MLStacks community to gather feedback on the Azure integration and make adjustments as necessary to address any issues or enhancements.

Additional Context

Expanding MLStacks to include Azure as a supported provider will significantly enhance the framework's versatility and appeal to users operating within the Azure ecosystem. This addition is a strategic move to ensure MLStacks remains a comprehensive and flexible solution for MLOps infrastructure deployment across all major cloud providers.

Code of Conduct

MASisserson commented 3 months ago

I'd like to take a stab at this, if that's alright.

strickvl commented 3 months ago

Sure thing! Let me know if you have questions along the way. I'd recommend starting small (just artifact store to start with) and then slowly adding new components incrementally. You might also draw some inspiration from these non-modular recipes from the 'old' mlstacks: https://github.com/zenml-io/mlstacks/tree/main/azure-minimal and https://github.com/zenml-io/mlstacks/tree/main/azureml-minimal

MASisserson commented 3 months ago

Thanks for the suggestions!