Add private Python packages to Enviroments in SDK v2

[rubenCrayon]

Is your feature request related to a problem? Please describe.

Adding private Python packages to Environments was possible using SDK v1 (https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-use-private-python-packages?view=azureml-api-1), while the option is not delivered in SDK v2. Apart from this, the documentation does not cover this functionality, so there is no way of creating from the existing version

Describe the solution you'd like

It would be helpful if the new Environment class from SDK v2 allows the user to generate environments with the possibility of adding private packages on it (before or after the rest of dependencies related in the conda.yml)

Describe alternatives you've considered

Using Environments from SDK v1 in SDK v2 cannot be performed, and building specific docker images for installing the private packages is highly more complicated that in the previous version

Additional context

[xiangyan99]

Thanks for the feedback, we’ll investigate asap.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github @Azure/azure-ml-sdk.

[SouravMalliK]

Looking forward for this resolution --

I too have the same question. Is there any way we could use the equivalent command using components using yaml then use these components to build pipeline.

whl_url = Environment.add_private_pip_wheel(workspace=ws,file_path = wheel_path1, exist_ok=True)

A similar issue is create here : https://github.com/Azure/azure-cli-extensions/issues/5111

[jrd232f]

Any news on this issue?

[astha29may]

Any update on this issue, or tentative timeline for the fix?

[SouravMalliK]

I found this documentation for SDK V2 using a new Class Model Package, which should server what we are looking for, provided we have a private python repository storing the private package --

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-package-models?view=azureml-api-2&tabs=sdk#package-a-model-that-has-dependencies-in-private-python-feeds

[jrd232f]

I found this documentation for SDK V2 using a new Class Model Package, which should server what we are looking for, provided we have a private python repository storing the private package --

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-package-models?view=azureml-api-2&tabs=sdk#package-a-model-that-has-dependencies-in-private-python-feeds

Awesome. Let me try that later. Thanks!

[AlexVialaBellander]

No update? Been searching the web and reading documentation for a week now. Cant find any good solution. Prob better to put the compute behind a vnet where your distribution server lies unprotected. But I hate networking and would rather have the possibility to set ENV variables to the conda creation compute.

[jinlow]

What we have done, that works, is host our private packages in an internal artifacts store (JFrog artifactory in our case), and then you can add a workspace connection to this private package repo. Then just add the line...

- pip:
  - --extra-index-url https://<Path to private artifactory>
  # Next, our internal packages, in addition to other common data science packages we want.
  - OUR_INTERNAL_PACKAGES
  - panads

Would this solve your solution? Or you are looking for a way without having to manage a pip repository?

[jinlow]

Actually, now I see this is the exact same solution that SouravMalliK mentioned!