[Errno 13] Permission denied

[MichielMosmans]

Hi,

I am trying to run just the stage 'Model CI' to create the aml pipeline. With an offline run the pipeline will be created succesfully and is displayed in the azure portal. However I encounter the following error when i try to run this stage with azure devops:

"main", mod_spec) File "/usr/local/envs/mlopspython_ci/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/w/1/s/ml_service/pipelines/build_train_pipeline.py", line 140, in main() File "/__w/1/s/ml_service/pipelines/build_train_pipeline.py", line 67, in main dataset = Dataset.Tabular.from_sql_query((sql_datastore, query)) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/data/_loggerfactory.py", line 78, in wrapper return func(*args, *kwargs) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/data/dataset_factory.py", line 378, in from_sql_query dataflow = dataprep().read_sql(_get_store_and_query(query)) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/readers.py", line 315, in read_sql return datastore_to_dataflow(DataPath(data_source, query)) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/_datastore_helper.py", line 46, in datastore_to_dataflow datastore, datastore_value = get_datastore_value(data_source) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/_datastore_helper.py", line 79, in get_datastore_value _set_auth_type(workspace) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/_datastore_helper.py", line 125, in _set_auth_type get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(AuthType.DERIVED, '')) File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/engineapi/api.py", line 18, in get_engine_api _engine_api = EngineAPI() File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/engineapi/api.py", line 52, in init self._message_channel = launch_engine() File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/dataprep/api/engineapi/engine.py", line 227, in launch_engine dependencies_path = runtime.ensure_dependencies() File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/dotnetcore2/runtime.py", line 116, in ensure_dependencies with _FileLock(deps_lock_path, raise_on_timeout=timeout_exception): File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/dotnetcore2/runtime.py", line 92, in enter__ self.acquire() File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/dotnetcore2/runtime.py", line 65, in acquire self.lockfile = os.open(self.lockfile_path, os.O_CREAT | os.O_EXCL | os.O_RDWR) PermissionError: [Errno 13] Permission denied: '/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/dotnetcore2/bin/deps.lock'

I am not sure if it is related to https://github.com/Microsoft/azure-pipelines-tasks/issues/8299 but I have not been successful in implementing a fix for this.

[dtzar]

Did you set all the required values for the environment variables in Azure DevOps or the *variables.yml file?

[MichielMosmans]

Did you set all the required values for the environment variables in Azure DevOps or the *variables.yml file?

Thank you for your response! This was indeed the problem. I changed the value for my local config but not for the devops config. Thanks again.

[gary918]

Hi, @dtzar, I'm having the same issue and I belive that I've set all the required values for the environment variables in Azure DevOps or the *variables.yml file. What are the required values? Should I input everything listed in env_variables.py? Any cluses why this error points to someting like ' raw_ds = Dataset.Tabular.from_delimited_files(path=raw_data_path) ' ?

[gary918]

@MichielMosmans just curious: have you added something related to the dataset and datafile?

[abij]

I ran into the same error.

Turns out that Dataprep uses dotnetcore2 to check and install libs from https://azuremldownloads.azureedge.net/dotnetcore2-dependencies/. It's done for specific Os versions. I noticed that Debian 10 (buster) was not available.

A lock before downloading is created in /usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/dotnetcore2/bin/deps.lock. Therefore the MLOps container is using a chmod -R 777 to grant write permissions.

I was creating a custom MLOps image and needed to use Debian v9 (stretch-slim) and allow write permissions to fix this issue. A suggestion is to pre-install these dependencies during the docker build of the MlOps image.

[dtzar]

FYI @j-so @eedorenko. If this is still a problem @abij you should file a new issue