Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.53k stars 2.76k forks source link

AzureML SDKV2 - ml_client job stream error - ManagedIdentityCredential authentication unavailable #35806

Closed obiii closed 2 months ago

obiii commented 3 months ago

Describe the bug We are submitting the aml job from devops, the job gets submitted but it raises exception when we try to stream it. This only happens when the job is submitted from the devops and not when we submit it from the compute manually.

We are using the Client ID of the user-assigned managed identity.

To Reproduce The code we use:

def get_chained_credentials(client_id: str):
    try:
        managed_identity_creds = ManagedIdentityCredential(client_id=client_id)
        default_creds = DefaultAzureCredential()
        chained_creds = ChainedTokenCredential(managed_identity_creds, default_creds)
        return chained_creds
   except Exception as e:
        raise e

ml_client = get_client_workspace(
    get_chained_credentials(e.cluster_identity_id),
    e.subscription_id,
    e.resource_group,
    e.workspace_name,
)
pipeline_run_job = ml_client.jobs.create_or_update(
    pipeline_job, experiment_name=e.experiment_name_forecast
)
ml_client.jobs.stream(name=pipeline_run_job.name)

Expected behavior If it is able to submit the job and the job completes as well, it should be able to stream the logs. We cant understand why it does so.

Screenshots image

github-actions[bot] commented 3 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

PratibhaShrivastav18 commented 2 months ago

@obiii Thanks for bringing this up. Couple of asks:

  1. Version of SDK you are using?
  2. Can you please help add the complete code (especially pipeline_job configuration).
  3. Does the compute you are using have the user assigned managed identity attached?
  4. Are you using any vnets/private networks?
PratibhaShrivastav18 commented 2 months ago

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!