search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.99k stars 2.97k forks source link

az login --identity fails on an AzureML compute instance: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #24675

Open leosannlsn opened 1 year ago

leosannlsn commented 1 year ago

Describe the bug

Command Name az login --identity

Errors:

The command failed with an unexpected error. Here is the traceback:
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/profile/custom.py", line 122, in login
    return profile.login_with_managed_identity(username, allow_no_subscriptions)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/_profile.py", line 234, in login_with_managed_identity
    msi_creds = MSIAuthenticationWrapper(resource=resource)
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__
    self.set_token()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
    super().set_token()
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 596, in set_token
    self.scheme, _, self.token = get_msi_token_webapp(self.resource, self.msi_conf)
  File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 548, in get_msi_token_webapp
    token_entry = result.json()
  File "/opt/az/lib/python3.10/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/opt/az/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/az/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/az/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Expected Behavior

az login --identity should log me in without failures

Environment Summary

Linux-5.15.0-1017-azure-x86_64-with-glibc2.31, Ubuntu 20.04.4 LTS
Python 3.10.5 ("base" conda environment)
Installer: DEB

azure-cli 2.39.0 *

Extensions:
ml 2.11.0

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

yonzhan commented 1 year ago

@jiasli for awareness

julyvee commented 1 year ago

This is still an issue for me and the solution to #20853 did not help in my case. This is my stack trace after using az login -i on an azure ml compute instance

@bebound @yonzhan @jiasli 

The command failed with an unexpected error. Here is the traceback:
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/anaconda/envs/azureml_py38/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/anaconda/envs/azureml_py38/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/command_modules/profile/custom.py", line 122, in login
    return profile.login_with_managed_identity(username, allow_no_subscriptions)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/_profile.py", line 234, in login_with_managed_identity
    msi_creds = MSIAuthenticationWrapper(resource=resource)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__
    self.set_token()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token
    super().set_token()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msrestazure/azure_active_directory.py", line 596, in set_token
    self.scheme, _, self.token = get_msi_token_webapp(self.resource, self.msi_conf)
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/msrestazure/azure_active_directory.py", line 548, in get_msi_token_webapp
    token_entry = result.json()
  File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
bebound commented 1 year ago

@julyvee your issue is similar to #23978

julyvee commented 1 year ago

@bebound Yes, but there is no real solution in that ticket. What does "Taking extra steps to login" even mean.

bebound commented 1 year ago

@julyvee jiasli send you an email, could you please take a look?

trojan03 commented 1 year ago

@bebound Could you please post the solution here? Having the same issue and have no clue what does "Taking extra steps to login" mean as well.

csm10495 commented 1 year ago

I hit this too just now. Exact same issue. Any steps to fix?

ghost commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Issue Details
## Describe the bug **Command Name** `az login --identity` **Errors:** ``` The command failed with an unexpected error. Here is the traceback: Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke cmd_result = self.invocation.execute(args) File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute raise ex File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job result = cmd_copy(params) File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__ return self.handler(*args, **kwargs) File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler return op(**command_args) File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/profile/custom.py", line 122, in login return profile.login_with_managed_identity(username, allow_no_subscriptions) File "/opt/az/lib/python3.10/site-packages/azure/cli/core/_profile.py", line 234, in login_with_managed_identity msi_creds = MSIAuthenticationWrapper(resource=resource) File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 592, in __init__ self.set_token() File "/opt/az/lib/python3.10/site-packages/azure/cli/core/auth/adal_authentication.py", line 75, in set_token super().set_token() File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 596, in set_token self.scheme, _, self.token = get_msi_token_webapp(self.resource, self.msi_conf) File "/opt/az/lib/python3.10/site-packages/msrestazure/azure_active_directory.py", line 548, in get_msi_token_webapp token_entry = result.json() File "/opt/az/lib/python3.10/site-packages/requests/models.py", line 910, in json return complexjson.loads(self.text, **kwargs) File "/opt/az/lib/python3.10/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/opt/az/lib/python3.10/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/az/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` ## To Reproduce: Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information. - Create a compute instance in AzureML (mine is STANDARD_DS11_V2) - Start the compute instance and click on the Terminal link to start a session - Run `az login --identity` as suggested by the welcome message ## Expected Behavior `az login --identity` should log me in without failures ## Environment Summary ``` Linux-5.15.0-1017-azure-x86_64-with-glibc2.31, Ubuntu 20.04.4 LTS Python 3.10.5 ("base" conda environment) Installer: DEB azure-cli 2.39.0 * Extensions: ml 2.11.0 Dependencies: msal 1.18.0b1 azure-mgmt-resource 21.1.0b1 ``` ## Additional Context
Author: leosannlsn
Assignees: -
Labels: `Account`, `Service Attention`, `Machine Learning`, `customer-reported`, `feature-request`, `Auto-Assign`, `Azure CLI Team`
Milestone: Backlog
jiasli commented 1 year ago

Routing to ML team to provide further guidance on what "Taking extra steps to login" is.

tunayokumus commented 1 year ago

Hi, we are also experiencing the same issue with the compute instances for most of the users in our AzureML workspaces. Interestingly for compute instances that are assigned to some other users (including my account), we are not able to reproduce this issue. @jiasli @azureml-github can you please share guidance on this? Happy to share further details to troubleshoot.

jwgwalton commented 1 year ago

Are we mean to assign a managed identity to compute instances? the notes on starting a terminal instance in Azure ML Studio are

Welcome to the Azure Machine Learning terminal

Enter "git clone [url]" to clone a repo Enter "git --help" to learn about Git CLI. To learn more about integrating Git with the Azure Machine Learning terminal, navigate here: https://learn.microsoft.com/en-us/azure/machine-learning/concept-train-model-git-integration#clone-git-repositories-into-your-workspace-file-system Enter "az ml --help" to learn about Azure ML CLI v2

Note: Use "az login --identity" instead of "az login" to avoid device code authentication

However there is no mention of the need for a managed identity.

NSimpragaVolur commented 1 year ago

Faced the same issue described here with az login --identity, even with creating the compute instance with a system-assigned identity. I managed to login using the identity by addind the --username flag and passing the managed identity's application ID, like this: az login --identity --username <application-id-from-aad-enterpriseapplication-for-the-managed-identity>

Question is - why does the ID need to be specified if it's a system-assigned managed identity tied to the compute instance?

natasha-savic-msft commented 1 year ago

@jiasli I have the same issue but when running cat ~/.azure/msal_token_cache.json (other thread, similar bug 20853) the file is not empty. The login fails when doing az login --identity

natasha-savic-msft commented 1 year ago

What solved the problem for me was to restart the compute instance and log in via az login.

NSimpragaVolur commented 1 year ago

What solved the problem for me was to restart the compute instance and log in via az login.

Does az login log you in with the system-assigned identity or with your own personal account?

Dave-Bouchard commented 1 year ago

Hi @julyvee, Did you solve this issue?

ant0nsc commented 1 year ago

We just faced the same issue. It seems that settings for identity-based auth had not been set up correctly, that's one thing. But also the fact that the compute instance had been created by another user caused problems.

There are two things that it took to resolve:

1) Copy settings for identity-based auth to environment variables: For that run while read -r line ; do export $line; done < /etc/environment.sso in your shell. This will copy settings for authentication endpoints.

2) The compute instance in question had been created by someone else for me. In AzureML Studio, go to the compute instance and click on "Jupyter" in the "Applications" column. This will bring up a warning message, asking you to click if you trust this compute instance. Click to confirm that.

Once both those steps had been done, I was able to run az login --identity correctly.

So in way I guess the error message that I received was masking that "do you trust this compute instance" problem.

astafan8 commented 1 year ago

Experienced this issue again on a brand new compute.

Then tried suggestion from this comment and step 2 didn't do anything (I didn't see the warning message) but step 1, saying run while read -r line ; do export $line; done < /etc/environment.sso in terminal, made az loging --identity work (not throw an error). However, whenever a new terminal is created this fix needs to be applied again.

I did notice that the az login -i said that i am now logged in with the identity of the virtual machine - is that expected/desired? I thought that all things in Azure ML are done via the user's identity. Note that for my use cases of azure.identity i need to loging into az as a user, not at the VM's identity.

@yonzhan, @jiasli, @bebound - i guess you're from the relevant team - could you please provide guidance on how things should be done? Also, could you propagate this information to the relevant Azure ML folks so that the need for this workaround can be removed?