Azure / MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
https://docs.microsoft.com/azure/machine-learning/service/
MIT License
4.07k stars 2.52k forks source link

(ServiceError) Received 403 from a service request when running an AzureML pipeline with AutoMLStep #1863

Closed nabil-fcbqa closed 8 months ago

nabil-fcbqa commented 1 year ago

I am trying to construct and run pipelines from AzureML using the Python SDK v1 from a Python script but run into the following error:

Traceback (most recent call last): File "pipeline.py", line 200, in <module> pipeline_run.wait_for_completion() File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/pipeline/core/run.py", line 289, in wait_for_completion for step_run in children: File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/run.py", line 467, in _rehydrate_runs for run_dto in run_dtos: File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py", line 425, in get_descendants for child in children: File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/experiment_client.py", line 342, in _client_filter for run_dto in run_dtos: File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 264, in _call_paginated_api paginated_dto = next_page.wait(awaiter_name="ApiPagination") File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_common/async_utils/async_task.py", line 58, in wait res = self._handler(self._future, self._logger) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_common/async_utils/async_task.py", line 16, in basic_handler return future.result() File "/anaconda/envs/azureml_py38/lib/python3.8/concurrent/futures/_base.py", line 439, in result return self.__get_result() File "/anaconda/envs/azureml_py38/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result raise self._exception File "/anaconda/envs/azureml_py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 333, in _execute_with_base_arguments return ClientBase._execute_func_internal( File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 367, in _execute_func_internal left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 399, in _handle_retry raise error File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/clientbase.py", line 358, in _execute_func_internal response = func(*args, **kwargs) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_restclient/operations/run_operations.py", line 107, in get_child raise models.ErrorResponseException(self._deserialize, response) azureml._restclient.models.error_response.ErrorResponseException: (ServiceError) Received 403 from a service request

The pipeline starts running just fine in the background but the terminal does not generate any logs following the error. In addition to that, the pipeline fails at the AutoMLStep for the following reason:

Received 403 from a service request-{
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "User is not authorized to access provided resources due to attempted access of the workspace data plane operations outside of the designated private link..",

These pipelines used to run fine until yesterday. Is there any change in behavior expected in the AzureML SDK v1? Thanks.