MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.2k stars 21.34k forks source link

How to instantiate MLClient pointing a registry inside a different region? #122537

Closed sudospyro closed 3 months ago

sudospyro commented 3 months ago

Using SDK azure-ai-ml v1.16.0

MLClient keeps throwing HttpResponseError due to unauthorised registry access when instanciated via:

MLClient(
            self.cred,
            registry_name=self.env.ml_registry_name,
            registry_location="westeurope"
        )

However I suspect this is due to the inability to configure the registry location.

azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'User/tenant/subscription is not allowed to access registry REGISTRY_NAME'
Content: {
  "error": {
    "code": "UserError",
    "severity": null,
    "message": "User/tenant/subscription is not allowed to access registry REGISTRY_NAME",
    "messageFormat": "User/tenant/subscription is not allowed to access registry {feedName}",
    "messageParameters": {
      "feedName": "REGISTRY_NAME"
    },
    "referenceCode": null,
    "detailsUri": null,
    "target": null,
    "details": [],
    "innerError": {
      "code": "Auth",
      "innerError": {
        "code": "Authorization",
        "innerError": {
          "code": "UnAuthorizedRegistryAccess",
          "innerError": null
        }
      }
    },
    "debugInfo": null,
    "additionalInfo": null
  },
  "correlation": {
    "operation": "******",
    "request": "*******"
  },
  "environment": "eastus",
  "location": "eastus",
  "time": "2024-05-15T15:47:29.07836+00:00",
  "componentName": "feedmanagement",
  "statusCode": 403
}

My registry, obfuscated with REGISTRY_NAME, resides inside westeurope, however this is querying eastus. Registries are unique per location, hence I doubt it is looking in the right place.

I have seen #105457 references a registry paramater, however this does not work as the underlying SDK does not make use of this anymore.

Documentation might be out of date now - docs relates to sharing environments with registries.

How can I instantiate MLClient pointing to my registry inside a different region?


Document Details

Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

PesalaPavan commented 3 months ago

@thedudetom Thanks for your feedback! We will investigate and update as appropriate.

sudospyro commented 3 months ago

I have now isolated this issue to when the component environment image lies inside the workspace acr: Note the distinction between ACR and registry. ACR represents the default ACR deployed in the workspace, registry represents the azureml registry resource.

To test this, I have created a new workspace, with a machine learning registry linked. Is there a permission I am missing?

Component definition:

$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command
version: 1

name: dummy
display_name: dummy

code: ./code

inputs:
  input1:
    type: string
    optional: true

environment: 
  image: WORKSPACE_ACR_NAME.azurecr.io/busybox:latest

command: >-
  python -V
cred = AzureCliCredential()
reg_ml_client = MLClient(cred, registry_name=AZUREML_REGISTRY_NAME)
component = load_component(COMPONENT_PATH)
reg_ml_client.components.create_or_update(component)
fkriti commented 3 months ago

@sudospyro Is your workspace in the same region? Also, could you please create a support ticket for this error and we can help investigate?

Naveenommi-MSFT commented 3 months ago

@sudospyro To gain a better understanding of your issue, I'd recommend working closer with our support team via an Azure support request.

sudospyro commented 3 months ago

Thanks @fkriti @Naveenommi-MSFT I have opened a support ticket. This issue was replicated in a new environment that was created via portal UI. All resources exist inside the same resource group, and region.

Error message returned from create_or_update()

azure.core.exceptions.HttpResponseError: (UserError) Authentication failed for container registry REGISTRY_NAME.azurecr.io
Code: UserError
Message: Authentication failed for container registry REGISTRY_NAME.azurecr.io

Running az acr login --name REGISTRY_NAME makes no difference to this error.

Naveenommi-MSFT commented 3 months ago

@sudospyro This issue could be due to incorrect registry authentication credentials or a network connectivity problem between the environment and the container registry. Please verify both.

We are going to close this thread, if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.