Update use private Python packages for v2

[rgreen1995]

Hi, I'm trying to move over to the v2 SDK, in particular, I'm looking to move my ACI and aks services to managed ml endpoints but I need to use some private packages in my environment and the current documentation does not apply to the v2 SDK. It would be good to update this page and/or add an equivalent page for v2

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: 8b4c6fed-3e7e-9ff8-c522-381b3c59b537
• Version Independent ID: 82bfe4a4-0f32-5adb-9820-c26dd2251906
• Content: Use private Python packages - Azure Machine Learning
• Content Source: articles/machine-learning/v1/how-to-use-private-python-packages.md
• Service: machine-learning
• Sub-service: core
• GitHub Login: @rastala
• Microsoft Alias: roastala

[ManoharLakkoju-MSFT]

@rgreen1995 Thanks for your feedback! We will investigate and update as appropriate.

[RamanathanChinnappan-MSFT]

@rgreen1995

Thanks for your feedback! We have assigned the issue to author and will provide further updates.

@rastala

could you please review this and update as appropriate.

[thomasfrederikhoeck]

Cross-linking with this issue https://github.com/Azure/azure-sdk-for-python/issues/22441

[hsuominen]

Any update on this?

[mark-feldmann]

Any news?

[robert4os]

Looking for this too

[jkyl]

this would be great to have! i would like to give the az ml environment create command access to my secrets, analogous to docker build --secret. alternately, maybe the build environment could be given access to the workspace's associated key vault. sadly, without support for this, the only way to use a private dependency in an Azure ML job is to build an image locally and push it to a container registry, then have the job's environment inherit from that image.

[jkyl]

There is some documentation here and here. I successfully created a connection to a private python feed using the steps suggested in the second link near the bottom of the notebook.

First I created the connection to my private PyPI-like index with a username and password credential that I know to work with normal pip/poetry/conda installations. I did not use a SasTokenConfiguration because my index (jfrog artifactory) does not support API tokens.

from azure.ai.ml.entities import WorkspaceConnection
from azure.ai.ml.entities import UsernamePasswordConfiguration

credentials = UsernamePasswordConfiguration(username=..., password=...)

ws_connection = WorkspaceConnection(
    name="<workspace_connection_name>",
    target="<python_feed_url>",
    type="python_feed",
    credentials=credentials,
)

ml_client.connections.create_or_update(ws_connection)

Then, I created an environment specification and a conda file that reference my private feed:

# env.yaml
$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: aml-private-feed
image: pytorch/pytorch:latest
conda_file: conda.yaml

# conda.yaml
name: aml-private-feed
channels:
  - defaults
dependencies:
  - python=3.10
  - pip
  - pip:
    - https://<my-private-feed>/<my-package>.whl

Finally, I submitted the environment build job to the same Azure ML workspace:

az ml environment create \
  --resource-group <my-resource-group> \
  --workspace-name <my-resource-group> \
  --subscription <my-subscription> \
  --file env.yaml

Judging from the build logs, Azure ML successfully mutated the dependency specification to insert my credentials into the https URL, and successfully downloaded the package from the private feed!

What I do not yet know is if this process will work for more generic builds, e.g. from a Docker context that picks up a pyproject.toml and poetry.lock. I will report back once I try that.

Update: this does not work with pyproject.toml and poetry.lock using a custom [[tool.poetry.source]] or a direct https URL dependency.

[thomasfrederikhoeck]

@jkyl We have gotten it to work for batch endpoints but not for online endpoints. See https://github.com/Azure/azure-sdk-for-python/issues/34336

[jkyl]

I would really love some documentation on how the dependency mutation works, what filetypes are supported, and what the roadmap is. Conda environment.yml files are really sub-optimal for a reliable build system, and a docker-based approach would tick all the boxes. docker build --secret seems to be an already-existing solution.

[mhaythornthwaite]

Any news on this?

[thomasfrederikhoeck]

I think this is solved, right? We use our Azure DevOps artifact which are used in the enviormenents when we build.

[rgreen1995]

I think this is solved, right? We use our Azure DevOps artifact which are used in the enviormenents when we build.

This solution works fine for me now, so happy to close the issue

[rgreen1995]

Also regarding the response above with it not working with poetry, I think this has changed now as I have it running fine with poetry and azure dev ops artifacts. I just had to make sure that I set the poetry config permissions with

poetry config http-basic.azure <username> <key>

then added the source with

poetry source add --priority=supplemental azure https://pkgs.dev.azure.com/<dev_ops_path>/pypi/simple

In the toml you should then see something like :

[[tool.poetry.source]] name = "azure" url = < dev_ops_url > priority = "supplemental"

after this you can either run

poetry add <package> --source azure

or in the toml under dependencies add

= {version = "^0.0.1", source = "azure"}