Can't create pipeline to execute notebook with custom environment

[hectordanielc]

• azureml-core:
• 1.55.1.post2:
• Linux:
• 3.10:

Describe the bug I'm trying to replicate this notebook https://github.com/balakreshnan/Samples2021/blob/main/AzureMLPipelineNotebook/notebookpipelineexec.md, but I can only use one curated environment. If I try to create an environment like in this code snipet (With USE_CURATED_ENV =False):

from azureml.core.runconfig import RunConfiguration
from azureml.core.conda_dependencies import CondaDependencies
from azureml.core import Environment 

aml_run_config = RunConfiguration()
#compute_target as defined in "Azure Machine Learning compute" section above
aml_run_config.target = compute_target

USE_CURATED_ENV = True
if USE_CURATED_ENV :
    curated_environment = Environment.get(workspace=ws, name="AzureML-Tutorial")
    aml_run_config.environment = curated_environment
else:
    aml_run_config.environment.python.user_managed_dependencies = False
    #Add some packages relied on by data prep step
    aml_run_config.environment.python.conda_dependencies = CondaDependencies.create(
        conda_packages=['pandas','scikit-learn','seaborn','tqdm'], 
        pip_packages=['azureml-sdk', 'azureml-dataprep[fuse,pandas]','seaborn','tqdm'], 
        pin_sdk_version=False)

the pipeline won't run and will return the error:

"This pipeline didn't have the rawdeserializer policy can't deserialize"

It only works with one curated environment that stats with AzurML and it doesn't matter if I clone it, the clone fails too and no other curated environment works neither.

The only curated environment that works doesn't have the required libraries so in order to use them I tried adding them, but adding libraries on it doesn't work neither, maybe because it doesn't have conda dependencies.

Something I noticed is the when I run: Environment.list(worspace=ws)

The only printed envrionment is the one that works, all the other available environments are not printed.

Expected behavior The pipeline should execute properly with every environment.

Screenshots

Additional context The trial.py its only a Hello World print.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

[Ramya-ML18]

Hi @ hectordanielc As per the Notebook USE_CURATED_ENV =True its showing that the AzureML-Tutorial is deprecated.

when I am using the USE_CURATED_ENV =False the default custom environment is created successfully and the pipeline jobs gets succeeded. and also please share the repro steps?

[github-actions[bot]]

Hi @hectordanielc. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[hectordanielc]

I'm exactly following the repo https://github.com/balakreshnan/Samples2021/blob/main/AzureMLPipelineNotebook/notebookpipelineexec.md and it works only with the Microsoft's curated environment "AzureML-ACPT-pytorch-1.13-py38-cuda11.7-gpu:10". All the others even the others curated environments aren't working.

I followed the error and here is the log:

[Ramya-ML18]

Hi @hectordanielc From the error message its saying that the Authentication error from the storage. Can you please check the permissions of the storage account whether storage account has the necessary permissions to access the workspace or not.

[github-actions[bot]]

Hi @hectordanielc. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @hectordanielc, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!