Additional help/documentation for new users [help wanted]

[Dekermanjian]

Hi, I am a new user of your plugin. First let me thank you for putting together such a wonderful tool. I have pretty good experience with Kedro and deploying models to on-prem servers.

I am new to deploying to Azure and I was hoping for some advice/best practices on deploying using Kedro-Azureml.

I am able to get my ML pipeline running using the Docker deployment method. I am trying to setup a CI/CD for this pipeline with Azure DevOps and am running into some difficulties. Do you setup a remote repository in Azure Devops and setup an Azure DevOps pipeline that triggers kedro azureml run on push to the main branch? Does kedro-azureml plugin generate a YAML file that can be used with Azure DevOps for setting up the pipeline?

Any advice would be greatly appreciated thank you.

[marrrcin]

Hi @Dekermanjian , thanks for your interest in our plugin!

• The plugin does not generate any Azure DevOps YAMLs.
• Usual flow with Azure DevOps is the following:
  1. Prepare azureml.yml config (e.g. in conf/azure folder) with oc.env interpolations for crucial parts related to the Azure configuration (see https://docs.kedro.org/en/latest/configuration/advanced_configuration.html#how-to-use-resolvers-in-the-omegaconfigloader)
  2. Build docker image with your code
  3. Run unit tests / other tests
  4. Push the docker image to ACR
  5. Run kedro azureml run from within the built docker container, like this:

     - task: AzureCLI@2
     inputs:
     azureSubscription: $(azureServiceConnection)
     scriptType: bash
     scriptLocation: inlineScript
     addSpnToEnvironment: true
     inlineScript: |
       echo "Launching the pipeline in Azure ML Workspace: $(AZURE_WORKSPACE_NAME)"
       docker run \
         -e AZURE_CLIENT_ID="$servicePrincipalId" \
         -e AZURE_CLIENT_SECRET="$servicePrincipalKey" \
         -e AZURE_TENANT_ID="$tenantId" \
         -e AZURE_STORAGE_ACCOUNT_KEY="$AML_STORAGE_KEY" \
         -e AZURE_COMPUTE_CLUSTER_NAME="$(AZURE_COMPUTE_CLUSTER_NAME)" \
         -e AZURE_STORAGE_CONTAINER="$(AZURE_STORAGE_CONTAINER)" \
         -e AZURE_STORAGE_ACCOUNT="$(AZURE_STORAGE_ACCOUNT)" \
         -e AZURE_WORKSPACE_NAME="$(AZURE_WORKSPACE_NAME)" \
         -e AZURE_SUBSCRIPTION_ID="$(AZURE_SUBSCRIPTION_ID)" \
         -e AZURE_RESOURCE_GROUP="$(AZURE_RESOURCE_GROUP)" \
         $(containerRegistry)/$(imageRepository):$(tag) kedro azureml -e azure run --pipeline __default__
     displayName: Run Kedro pipeline in Azure ML
     env:
     AML_STORAGE_KEY: $(AZURE_STORAGE_ACCOUNT_KEY) # https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables

[Dekermanjian]

@marrrcin Thank you very much for your detailed response!

[marrrcin]

Can I make the issue as resolved?

[Dekermanjian]

Yes, I am sorry I thought I closed the issue with my comment.