microsoft / MLOpsPython

MLOps using Azure ML Services and Azure DevOps
MIT License
1.2k stars 1.1k forks source link

How to use azure devops artifacts repository as source for DatabricksStep of AzureML? #342

Open anirbansaha96 opened 3 years ago

anirbansaha96 commented 3 years ago

If we have PyPi Packages added as Artifacts to an Azure DevOps Project Feed, how can we use these packages as a source for installing packages in DatabricksStep of Azure Machine Learning Service?

While using pip in any environment, we use our Azure DevOps Project Artifacts feed in the following way:

pip install example-package --index-url=https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/

The DatabricksStep class of the Azure Machine Learning Service accepts the following parameters:

python_script_name = "<Some-Script>.py"
source_directory = "<Path-To-Script>"

<Some-Placeholder-Name-for-the-step> = DatabricksStep(
    name=<Some-Placeholder-Name-for-the-step>,
    num_workers=1,
    python_script_name=python_script_name,
    source_directory=source_directory,
    run_name= <Name-of-the-run>,
    compute_target=databricks_compute,
    pypi_libraries = [
                      PyPiLibrary(package = 'scikit-learn'), 
                      PyPiLibrary(package = 'scipy'), 
                      PyPiLibrary(package = 'azureml-sdk'), 
                      PyPiLibrary(package = 'joblib'), 
                      PyPiLibrary(package = 'azureml-dataprep[pandas]'),
                      PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/')
                    ], 

    allow_reuse=True
)

However, PyPiLibrary(package = 'example-package', repo='https://<Personal-Access-Token>@pkgs.dev.azure.com/<Organization-Name>/_packaging/<Artifacts-Feed-Name>/pypi/simple/') will give an error. How exactly should we consume the Artifacts Feed as an input to the PyPiLibrary property of the DatabricksStep Class in Azure Machine Learning Service?