Report on the issue of not being able to pass multiple arguments to the `docker_args` argument of the `azure.ai.ml.command` function in Azure Machine Learning SDK V2

[KazuOnuki]

Hi, All. This is a report of an issue by Azure Machine Learning Support Engineer. In the command function of SDK V2, docker_args argument type is a string and multiple arguments cannot be passed.

command

The sample code to reproduce the event is as follows

• Sample Code

  # Problem sample
  job = command(
  command='df -h && sleep 30 && exit',
  environment='AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:9',
  compute='<Compute Name>', #Compute instance
  docker_args='--cap-add=SYS_ADMIN --device=/dev/fuse --security-opt apparmor:unconfined'
  )
  # submit the command
  returned_job = ml_client.jobs.create_or_update(job)

• Error Message

  Failed to start Docker container xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx(masking)-execution-wrapper due to: Invalid container capability (cap-add): SYS_ADMIN --device=/dev/fuse --security-opt apparmor:unconfined. Supported values: ALL, AUDIT_CONTROL, AUDIT_READ, AUDIT_WRITE, BLOCK_SUSPEND, BPF, CHECKPOINT_RESTORE, CHOWN, DAC_OVERRIDE, DAC_READ_SEARCH, FOWNER, FSETID, IPC_LOCK, IPC_OWNER, KILL, LEASE, LINUX_IMMUTABLE, MAC_ADMIN, MAC_OVERRIDE, MKNOD, NET_ADMIN, NET_BIND_SERVICE, NET_BROADCAST, NET_RAW, PERFMON, SETFCAP, SETGID, SETPCAP, SETUID, SYS_ADMIN, SYS_BOOT, SYS_CHROOT, SYS_MODULE, SYS_NICE, SYS_PACCT, SYS_PTRACE, SYS_RAWIO, SYS_RESOURCE, SYS_TIME, SYS_TTY_CONFIG, SYSLOG, WAKE_ALARM. This error may occur if invalid or unsupported values are passed in the Docker arguments (for example with `--cap-add` ou `--cap-drop`).

  As a result of discussions with the Azure Machine Learning Product Group, it has been determined that the error occurs because the command function in the following public information is implemented as a string, whereas it is passed as a list. I assume that the correction of the implementation of the command function is being checked internally, but I would appreciate it if you could also correct this document.

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: f97f70b2-cf01-935c-3518-8476907c04be
• Version Independent ID: 455fd1a3-5529-efb9-8a36-63d43e45d6b8
• Content: azure.ai.ml package
• Content Source: docs-ref-autogen/azure-ai-ml/azure.ai.ml.yml
• Service: machine-learning
• GitHub Login: @lmazuel
• Microsoft Alias: lmazuel

[github-actions[bot]]

Thank you for your feedback. This has been routed to the support team for assistance.

[iscai-msft]

Thank you for opening this issue @KazuOnuki, tagging @azureml-github to take a look. thanks!

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github @Azure/azure-ml-sdk.

[smurli]

+1 any solution for this issue?

[rogeriobonatti]

@iscai-msft +1 any solution? I don't get an error when sending the docker_args, but it looks like they are not really taking any effect on the container creation. For example when I use: docker_args="-p=8006:8006 --device=/dev/kvm --cap-add=NET_ADMIN --stop-timeout=120", I get errors in the rest of my script because there runtime args were not correctly set

[iscai-msft]

@luigiw can you take a look at this issue? there are multiple people running into this error. Thanks!

[jhrmnn]

@iscai-msft, I'm leading a team at MSR and we're also blocked by this.

[iscai-msft]

@achauhan-scc are you on the ML team and starting work on this issue? There are a lot of people blocked by this issue so fast resolution would be appreciated. Thank you!

[achauhan-scc]

@iscai-msft - Yes, I am from AML team and actively looking into the issue, we are targeting a fix in Nov release but before that we need to close some of reviews as it's a very long pending issue (which is taking longer due to many engineers moved around)