Open jcormier opened 3 years ago
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @toddysm, @yugangw-MSFT.
Author: | jcormier |
---|---|
Assignees: | - |
Labels: | `Container Registry`, `Service Attention` |
Milestone: | - |
acr
I have exactly the same issue here. It hangs at this stage:
urllib3.connectionpool : Starting new HTTPS connection (1): technaturecontainerregistry.azurecr.io:443
urllib3.connectionpool : https://technaturecontainerregistry.azurecr.io:443 "POST /oauth2/exchange HTTP/1.1" 200 None
same issue. hangs at this point
I had the same issue, then opened a support request on Azure, then an Indian MS Support engineer contacted me, and we resolved the issue. Supposedly they changed something in Azure.
CC @adewaleo to cross check the registry's log for the error from @teusbenschop. The error from @mikeclearbank appears from ARM which is not ACR
CC @adewaleo to cross check the registry's log for the error from @teusbenschop. The error from @mikeclearbank appears from ARM which is not ACR
You are correct, the issue from @mikeclearbank differs from mine.
This bug hasn't seen activity in over a month, but the issue is still occurring; same exact symptom as the original bug filer's. Except, I think have some additional information. In all cases where I've let the command run, it does eventually complete, and in all the cases that I've let it run, it hangs for almost exactly 10 minutes.
urllib3.connectionpool : Starting new HTTPS connection (1): <ACR>.azurecr.io:443
urllib3.connectionpool : https://<ACR>.azurecr.io:443 "POST /oauth2/exchange HTTP/1.1" 200 None
<HANG for 10 mintues>
Login Succeeded
Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x1078ec4c0>, <function _x509_from_base64_to_hex_transform at 0x1078ec550>]
In this example, it was 6:39:11 PM when the line before "Login Succeeded" was emitted, and 6:49:12 PM when "Login Succeeded" was emitted.
A screencap from a different instance of this:
The two ACRs are masked, but they're also not the same. (That is, we're not seeing this behavior against a single, specific ACR instance.)
@roy-work, could you please contact me at yugangw at microsoft dot com, or if you don't mind, you can share the register name here, which I need to cross check the service log. I assume the time you provided are UTC or let me know the timezone?
@yugangw-msft I've sent you an email with those details.
(Relaying part of the email conversation here, in the hopes that it'll help folks debug this issue if they run into it.)
A working theory on this is that acr login
is doing some sort of docker
call/command during the hang.
In the course of debugging something completely separate, I discovered that Docker for OS X on my machine was hung; the whale icon in the menu was grey, and hovering over it got me the beachball. It occurs to me (since I don't know how long that was going on) that it could be related to this; perhaps a recent Docker update has introduced some bug recently that causes hangs, and that also translates to hangs inside acr login
, and that's just how I'm noticing it.
It sounds like acr login
is doing something docker related at this point. (If you read the logs posted here, you see that the POST /oauth2/exchange
call appears to have completed, but that raises the question: if that finished, then what are we waiting on?) I've asked that acr login
log whatever it is that it is doing here, since then it would provide clear feedback (at least, if you're using --debug
) as to what the hang was for.
Thanks @roy-work for discovering one of the potential causes. I have submitted https://github.com/Azure/azure-cli/pull/17115 to output more trace for self diagnosis
I'm experiencing the same issue - the command hangs.
az acr login --name acr0finall0dockers --debug
cli.knack.cli: Command arguments: ['acr', 'login', '--name', 'acr0finall0dockers', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x103c908b0>, <function OutputProducer.on_global_arguments at 0x103cf5af0>, <function CLIQuery.on_global_arguments at 0x103d18310>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'acr': ['azure.cli.command_modules.acr']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: acr 0.010 27 113
cli.azure.cli.core: Total (1) 0.010 27 113
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: Loaded 27 groups, 113 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : acr login
cli.azure.cli.core: Command table: acr login
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x1044bfe50>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/maciej.misztal/.azure/commands/2021-08-02.11-58-52.acr_login.51410.log'.
az_command_data_logger: command args: acr login --name {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x1044e1310>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x104544d30>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x104544dc0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x104544ee0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x103cf5b80>, <function CLIQuery.handle_query_parameter at 0x103d183a0>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x104544ca0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x104544e50>]
az --version
azure-cli 2.26.1
core 2.26.1
telemetry 1.0.6
Extensions:
azure-devops 0.18.0
Python location '/usr/local/Cellar/azure-cli/2.26.1/libexec/bin/python'
Extensions directory '/Users/maciej.misztal/.azure/cliextensions'
Python (Darwin) 3.8.11 (default, Jun 29 2021, 03:08:07)
[Clang 12.0.5 (clang-1205.0.22.9)]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Please let us know how we are doing: https://aka.ms/azureclihats
and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
acr service team should take a look
just noticed that my local docker was stuck in booting up state - I've reset it, and then it worked
Can you please prioritize this? Several years has past and no progress... As far as I know acr is paid service.
Getting the same issue when repo is not found:
^C% (aivanou-dev-new) ➜ ~ az acr login --name aivanuoreg --verbose --debug
cli.knack.cli: Command arguments: ['acr', 'login', '--name', 'aivanuoreg', '--verbose', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f7eb48cd2d0>, <function OutputProducer.on_global_arguments at 0x7f7eb4a004c0>, <function CLIQuery.on_global_arguments at 0x7f7eb4a35900>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'acr': ['azure.cli.command_modules.acr']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: acr 0.009 30 126
cli.azure.cli.core: Total (1) 0.009 30 126
cli.azure.cli.core: Loaded 30 groups, 126 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : acr login
cli.azure.cli.core: Command table: acr login
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f7eb573e170>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/aivanou/.azure/commands/2022-05-30.18-59-43.acr_login.50461.log'.
az_command_data_logger: command args: acr login --name {} --verbose --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f7eb574f370>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f7eb57a3a30>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f7eb57a3b50>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f7eb4a00550>, <function CLIQuery.handle_query_parameter at 0x7f7eb4a35990>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f7eb57a3ac0>]
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ResourceManagementClient
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/aivanou/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /Users/aivanou/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/8ac310c8-26a8-46a3-807b-9fc844be91bf/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 6a1b00f1-dc38-42b0-b027-45db861da9b0
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/0afcfdf9-426d-432e-8053-ece38c7f4209/resources?$filter=resourceType%20eq%20%27Microsoft.ContainerRegistry%2Fregistries%27&api-version=2021-04-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '56ee3090-e085-11ec-93c2-186590de3dbb'
cli.azure.cli.core.sdk.policies: 'CommandName': 'acr login'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--name --verbose --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.37.0 (PIP) azsdk-python-azure-mgmt-resource/21.1.0b1 Python/3.10.4 (macOS-10.16-x86_64-i386-64bit)'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/0afcfdf9-426d-432e-8053-ece38c7f4209/resources?$filter=resourceType%20eq%20%27Microsoft.ContainerRegistry%2Fregistries%27&api-version=2021-04-01 HTTP/1.1" 200 424
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Content-Encoding': 'gzip'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'Vary': 'Accept-Encoding'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11998'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '55810724-4ce5-41b9-b1b4-dda466e33a42'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '55810724-4ce5-41b9-b1b4-dda466e33a42'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTCENTRALUS:20220531T015944Z:55810724-4ce5-41b9-b1b4-dda466e33a42'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Date': 'Tue, 31 May 2022 01:59:43 GMT'
cli.azure.cli.core.sdk.policies: 'Content-Length': '424'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"value":[{"id":"/subscriptions/0afcfdf9-426d-432e-8053-ece38c7f4209/resourceGroups/aivanou-test/providers/Microsoft.ContainerRegistry/registries/aivanoureg","name":"aivanoureg","type":"Microsoft.ContainerRegistry/registries","sku":{"name":"Standard","tier":"Standard"},"location":"eastus","tags":{},"systemData":{"createdBy":"tierex02@gmail.com","createdByType":"User","createdAt":"2022-05-31T01:54:44.2842966Z","lastModifiedBy":"tierex02@gmail.com","lastModifiedByType":"User","lastModifiedAt":"2022-05-31T01:54:44.2842966Z"}}]}
cli.azure.cli.command_modules.acr._docker_utils: Could not get registry from service. Exception: The resource with name 'aivanuoreg' and type 'Microsoft.ContainerRegistry/registries' could not be found in subscription 'Azure subscription 1 (0afcfdf9-426d-432e-8053-ece38c7f4209)'.
urllib3.connectionpool: Starting new HTTPS connection (1): aivanuoreg.azurecr.io:443
I had the same issue, I was able to solve this by doing the following:
az login
with the proper tenant for my ACR instanceNote: I'm running wsl2 with Ubuntu 22.04 on Windows 11. My az cli version is:
azure-cli 2.40.0
core 2.40.0
telemetry 1.0.8
Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1
Python location '/opt/az/bin/python3'
Extensions directory '/home/seankane/.azure/cliextensions'
Python (Linux) 3.10.5 (main, Sep 2 2022, 05:41:19) [GCC 11.2.0]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Having the same issue. Other suggestions here are not working. I've tried restarting docker desktop and re-running az login
.
Deleting the ~/.docker/config.json file worked for me. https://stackoverflow.com/questions/72645895/credential-saving-error-when-connecting-to-service-on-docker
Still an issue
Restarting Docker Desktop, then running az acr login
again worked for me on Pop!_OS 22.04 LTS
If restarting Docker Desktop and running az acr login
does not work, you may also need to update your Docker Desktop client. I had the same issue and this appeared to resolve it.
I've had this issue. It seems to happen when Docker Desktop is in "Resource Saver" mode from inactivity. Restarting Docker Desktop, or running any image should fix the issue.
Tried everything on here except for contacting MS. I cant, I am just an employee, I don't run our subscriptions. Running on WSL. This was working a month again now its not. I am 90% sure this is a corporate proxy issue or something in between. Any updates on this?
Describe the bug Running
az acr login --name ensuretouchdev
never finishesTo Reproduce Run command
Expected behavior Login into registry
Environment summary
Ubuntu 18.04
Additional context