search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.61k stars 2.82k forks source link

Authentication fails only when debugging a pytest. #38094

Open lovettchris opened 1 week ago

lovettchris commented 1 week ago

azure-common 1.1.28 azure-core 1.31.0 azure-data-tables 12.5.0 azure-identity 1.19.0 azure-keyvault-keys 4.9.0 azure-keyvault-secrets 4.8.0 azure-mgmt-compute 33.0.0 azure-mgmt-core 1.4.0 azure-storage-blob 12.23.1

Describe the bug I have a simple Azure blob that I'm trying to connect to using DefaultCredentials in a pytest and it works fine when I run pytest from the command line, but when I try and debug the pytest in VSCode it fails with strange errors.

To Reproduce Steps to reproduce the behavior:

  1. debug a pytest that connects to Azure blob store.

Expected behavior should work the same

Screenshots

Additional context

Here's the debug log:

d:\git\athens\SmartReplay\tests\agents\test_agent_utils.py::test_setup_model failed: command = 'az account get-access-token --output json --resource https://storage.azure.com'
timeout = 10

    def _run_command(command: str, timeout: int) -> str:
        # Ensure executable exists in PATH first. This avoids a subprocess call that would fail anyway.
        if shutil.which(EXECUTABLE_NAME) is None:
            raise CredentialUnavailableError(message=CLI_NOT_FOUND)

        if sys.platform.startswith("win"):
            args = ["cmd", "/c", command]
        else:
            args = ["/bin/sh", "-c", command]
        try:
            working_directory = get_safe_working_dir()

            kwargs: Dict[str, Any] = {
                "stderr": subprocess.PIPE,
                "stdin": subprocess.DEVNULL,
                "cwd": working_directory,
                "universal_newlines": True,
                "timeout": timeout,
                "env": dict(os.environ, AZURE_CORE_NO_COLOR="true"),
            }
>           return subprocess.check_output(args, **kwargs)

D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

timeout = 10
popenargs = (['cmd', '/c', 'az account get-access-token --output json --resource https://storage.azure.com'],)
kwargs = {'cwd': 'C:\\WINDOWS', 'env': {'ADA_GIT_REPO': 'https://github.com/microsoft/ada', 'ADA_STORAGE_CONNECTION_STRING': 'D...EEwpKDuE2QeqC1BUfTdn/9o8MQY=;Version=1.0;', 'ALLUSERSPROFILE': 'C:\\ProgramData', ...}, 'stderr': -1, 'stdin': -3, ...}

    def check_output(*popenargs, timeout=None, **kwargs):
        r"""Run command with arguments and return its output.

        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.

        The arguments are the same as for the Popen constructor.  Example:

        >>> check_output(["ls", "-l", "/dev/null"])
        b'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.

        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        b'ls: non_existent_file: No such file or directory\n'

        There is an additional optional argument, "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it too will be used internally.  Example:

        >>> check_output(["sed", "-e", "s/foo/bar/"],
        ...              input=b"when in the course of fooman events\n")
        b'when in the course of barman events\n'

        By default, all communication is in bytes, and therefore any "input"
        should be bytes, and the return value will be bytes.  If in text mode,
        any "input" should be a string, and the return value will be a string
        decoded according to locale encoding, or by "encoding" if set. Text mode
        is triggered by setting any of text, encoding, errors or universal_newlines.
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')

        if 'input' in kwargs and kwargs['input'] is None:
            # Explicitly passing input=None was previously equivalent to passing an
            # empty string. That is maintained here for backwards compatibility.
            if kwargs.get('universal_newlines') or kwargs.get('text') or kwargs.get('encoding') \
                    or kwargs.get('errors'):
                empty = ''
            else:
                empty = b''
            kwargs['input'] = empty

>       return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                   **kwargs).stdout

D:\Anaconda3\envs\sr\lib\subprocess.py:421: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = 10, check = True
popenargs = (['cmd', '/c', 'az account get-access-token --output json --resource https://storage.azure.com'],)
kwargs = {'cwd': 'C:\\WINDOWS', 'env': {'ADA_GIT_REPO': 'https://github.com/microsoft/ada', 'ADA_STORAGE_CONNECTION_STRING': 'D...EEwpKDuE2QeqC1BUfTdn/9o8MQY=;Version=1.0;', 'ALLUSERSPROFILE': 'C:\\ProgramData', ...}, 'stderr': -1, 'stdin': -3, ...}
process = <Popen: returncode: 1 args: ['cmd', '/c', 'az account get-access-token --out...>
stdout = ''
stderr = "'az' is not recognized as an internal or external command,\noperable program or batch file.\n"
retcode = 1

    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://storage.azure.com']' returned non-zero exit status 1.

D:\Anaconda3\envs\sr\lib\subprocess.py:526: CalledProcessError

The above exception was the direct cause of the following exception:

    def test_setup_model():

>       store = get_store(blob_container_name=BLOB_CONTAINER_NAME, table_name=TABLE_NAME)

tests\agents\test_agent_utils.py:18: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
smart_replay\store\store.py:36: in get_store
    return AzureStore(account_id, blob_container_name=blob_container_name, table_name=table_name)
smart_replay\store\store.py:104: in __init__
    self.container_handler = AzureContainerHandler(storage_account_id, self.blob_container_name)
smart_replay\store\storage_handler.py:122: in __init__
    self.container_client = self._create_container_client()
smart_replay\store\storage_handler.py:138: in _create_container_client
    if not client.exists():
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\tracing\decorator.py:94: in wrapper_use_tracer
    return func(*args, **kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_container_client.py:554: in exists
    process_storage_error(error)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_shared\response_handlers.py:92: in process_storage_error
    raise storage_error
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_container_client.py:550: in exists
    self._client.container.get_properties(**kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\tracing\decorator.py:94: in wrapper_use_tracer
    return func(*args, **kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_generated\operations\_container_operations.py:1063: in get_properties
    pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:229: in run
    return first_node.send(pipeline_request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_redirect.py:197: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_shared\policies.py:556: in send
    raise err
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_shared\policies.py:528: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_authentication.py:145: in send
    self.on_request(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_authentication.py:120: in on_request
    self._request_token(*self._scopes)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_authentication.py:94: in _request_token
    self._token = cast(SupportsTokenInfo, self._credential).get_token_info(*scopes, options=options)
D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_internal\decorators.py:23: in wrapper
    token = fn(*args, **kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:125: in get_token_info
    return self._get_token_base(*scopes, options=options)
D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:147: in _get_token_base
    output = _run_command(command, self._process_timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

command = 'az account get-access-token --output json --resource https://storage.azure.com'
timeout = 10

    def _run_command(command: str, timeout: int) -> str:
        # Ensure executable exists in PATH first. This avoids a subprocess call that would fail anyway.
        if shutil.which(EXECUTABLE_NAME) is None:
            raise CredentialUnavailableError(message=CLI_NOT_FOUND)

        if sys.platform.startswith("win"):
            args = ["cmd", "/c", command]
        else:
            args = ["/bin/sh", "-c", command]
        try:
            working_directory = get_safe_working_dir()

            kwargs: Dict[str, Any] = {
                "stderr": subprocess.PIPE,
                "stdin": subprocess.DEVNULL,
                "cwd": working_directory,
                "universal_newlines": True,
                "timeout": timeout,
                "env": dict(os.environ, AZURE_CORE_NO_COLOR="true"),
            }
            return subprocess.check_output(args, **kwargs)
        except subprocess.CalledProcessError as ex:
            # non-zero return from shell
            # Fallback check in case the executable is not found while executing subprocess.
            if ex.returncode == 127 or ex.stderr.startswith("'az' is not recognized"):
>               raise CredentialUnavailableError(message=CLI_NOT_FOUND) from ex
E               azure.identity._exceptions.CredentialUnavailableError: Azure CLI not found on path

D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:239: CredentialUnavailableError
pvaneck commented 1 week ago

From the error message, the Azure CLI executable az can't be found in the path. Perhaps when you run or debug tests in VS Code, it might be using a different environment that doesn't have the Azure CLI in its PATH.

What terminal does your VS Code use to run tests, and is it the same as the normal terminal you used to run the tests manually?

If you print os.environ.get("PATH") in your test, you can also check to see how it differs in VS Code versus normally. Ultimately, I think this is a VS Code configuration issue and some adjustments might be needed.

github-actions[bot] commented 1 week ago

Hi @lovettchris. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

lovettchris commented 1 week ago

Ok, I made sure az is in my PATH for the USER and for SYSTEM and not just for the command line environment, and I get the same error. Here's a small repro:

from azure.identity import AzureCliCredential
from azure.storage.blob import BlobServiceClient

def test_azure():
    credentials = AzureCliCredential()
    storage_account = "srexperiments"
    blob_container_name = "unittests"
    account_url = f"https://{storage_account}.blob.core.windows.net/"
    blob_service_client = BlobServiceClient(account_url, credential=credentials, logging_enable=False)
    container_client = blob_service_client.get_container_client(container=blob_container_name)
    assert container_client.exists()

Steps

  1. Create storage account "srexperiments" and blob container "unittests" and make sure your SC-ALT account has these permissions:

    • "Reader",
    • "Storage Blob Data Reader",
    • "Storage Blob Data Contributor",
    • "Storage Queue Data Reader",
    • "Storage Queue Data Contributor",
    • "Storage Table Data Reader",
    • "Storage Table Data Contributor",
    • "Storage File Data Privileged Reader",

  2. Install azcopy and az cli and make sure az is in your user PATH and system PATH.

  3. Use azcopy login and login to your Azure SC-ALT account using web browser.

  4. Use az login to also login to the same account (should be quick because of cached credentials.

  5. run pytest from command line (works fine)

  6. debug the pytest in VS code, fails with the following errors:

Running pytest with args: ['-p', 'vscode_pytest', '--rootdir=d:\\temp\\test', '--capture=no', 'd:\\temp\\test\\test_azure.py::test_azure']
============================= test session starts =============================
platform win32 -- Python 3.10.15, pytest-7.3.1, pluggy-1.5.0
rootdir: d:\temp\test
plugins: anyio-4.6.2.post1
collected 1 item

test_azure.py F

================================== FAILURES ===================================
_________________________________ test_azure __________________________________
command = 'az account get-access-token --output json --resource https://storage.azure.com'
timeout = 10
    def _run_command(command: str, timeout: int) -> str:
        # Ensure executable exists in PATH first. This avoids a subprocess call that would fail anyway.
        if shutil.which(EXECUTABLE_NAME) is None:
            raise CredentialUnavailableError(message=CLI_NOT_FOUND)

        if sys.platform.startswith("win"):
            args = ["cmd", "/c", command]
        else:
            args = ["/bin/sh", "-c", command]
        try:
            working_directory = get_safe_working_dir()

            kwargs: Dict[str, Any] = {
                "stderr": subprocess.PIPE,
                "stdin": subprocess.DEVNULL,
                "cwd": working_directory,
                "universal_newlines": True,
                "timeout": timeout,
                "env": dict(os.environ, AZURE_CORE_NO_COLOR="true"),
            }
>           return subprocess.check_output(args, **kwargs)

D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:234:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
timeout = 10
popenargs = (['cmd', '/c', 'az account get-access-token --output json --resource https://storage.azure.com'],)
kwargs = {'cwd': 'C:\\WINDOWS', 'env': {'ADA_GIT_REPO': 'https://github.com/microsoft/ada', 'ADA_STORAGE_CONNECTION_STRING': 'D...EEwpKDuE2QeqC1BUfTdn/9o8MQY=;Version=1.0;', 'ALLUSERSPROFILE': 'C:\\ProgramData', ...}, 'stderr': -1, 'stdin': -3, ...}
    def check_output(*popenargs, timeout=None, **kwargs):
        r"""Run command with arguments and return its output.

        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.

        The arguments are the same as for the Popen constructor.  Example:

        >>> check_output(["ls", "-l", "/dev/null"])
        b'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.

        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        b'ls: non_existent_file: No such file or directory\n'

        There is an additional optional argument, "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it too will be used internally.  Example:
        >>> check_output(["sed", "-e", "s/foo/bar/"],
        ...              input=b"when in the course of fooman events\n")
        b'when in the course of barman events\n'

        By default, all communication is in bytes, and therefore any "input"
        should be bytes, and the return value will be bytes.  If in text mode,
        any "input" should be a string, and the return value will be a string
        decoded according to locale encoding, or by "encoding" if set. Text mode
        is triggered by setting any of text, encoding, errors or universal_newlines.
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')

        if 'input' in kwargs and kwargs['input'] is None:
            # Explicitly passing input=None was previously equivalent to passing an
            # empty string. That is maintained here for backwards compatibility.
            if kwargs.get('universal_newlines') or kwargs.get('text') or kwargs.get('encoding') \
                    or kwargs.get('errors'):
                empty = ''
            else:
                empty = b''
            kwargs['input'] = empty

>       return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                   **kwargs).stdout
D:\Anaconda3\envs\sr\lib\subprocess.py:421: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, capture_output = False, timeout = 10, check = True
popenargs = (['cmd', '/c', 'az account get-access-token --output json --resource https://storage.azure.com'],)
kwargs = {'cwd': 'C:\\WINDOWS', 'env': {'ADA_GIT_REPO': 'https://github.com/microsoft/ada', 'ADA_STORAGE_CONNECTION_STRING': 'D...EEwpKDuE2QeqC1BUfTdn/9o8MQY=;Version=1.0;', 'ALLUSERSPROFILE': 'C:\\ProgramData', ...}, 'stderr': -1, 'stdin': -3, ...}
process = <Popen: returncode: 1 args: ['cmd', '/c', 'az account get-access-token --out...>
stdout = ''
stderr = "'az' is not recognized as an internal or external command,\noperable program or batch file.\n"
retcode = 1
    def run(*popenargs,
            input=None, capture_output=False, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.

        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them,
        or pass capture_output=True to capture both.

        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.

        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.

        There is an optional argument "input", allowing you to
        pass bytes or a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.

        By default, all communication is in bytes, and therefore any "input" should
        be bytes, and the stdout and stderr will be bytes. If in text mode, any
        "input" should be a string, and stdout and stderr will be strings decoded
        according to locale encoding, or by "encoding" if set. Text mode is
        triggered by setting any of text, encoding, errors or universal_newlines.

        The other arguments are the same as for the Popen constructor.
        """
        if input is not None:
            if kwargs.get('stdin') is not None:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE

        if capture_output:
            if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
                raise ValueError('stdout and stderr arguments may not be used '
                                 'with capture_output.')
            kwargs['stdout'] = PIPE
            kwargs['stderr'] = PIPE

        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired as exc:
                process.kill()
                if _mswindows:
                    # Windows accumulates the output in a single blocking
                    # read() call run on child threads, with the timeout
                    # being done in a join() on those threads.  communicate()
                    # _after_ kill() is required to collect that and add it
                    # to the exception.
                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['cmd', '/c', 'az account get-access-token --output json --resource https://storage.azure.com']' returned non-zero exit status 1.
D:\Anaconda3\envs\sr\lib\subprocess.py:526: CalledProcessError

The above exception was the direct cause of the following exception:

    def test_azure():
        credentials = AzureCliCredential()
        storage_account = "srexperiments"
        blob_container_name = "unittests"
        account_url = f"https://{storage_account}.blob.core.windows.net/"
        blob_service_client = BlobServiceClient(account_url, credential=credentials, logging_enable=False)
        container_client = blob_service_client.get_container_client(container=blob_container_name)
>       assert container_client.exists()

test_azure.py:12:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\tracing\decorator.py:94: in wrapper_use_tracer
    return func(*args, **kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_container_client.py:554: in exists
    process_storage_error(error)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_shared\response_handlers.py:92: in process_storage_error
    raise storage_error
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_container_client.py:550: in exists
    self._client.container.get_properties(**kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\tracing\decorator.py:94: in wrapper_use_tracer
    return func(*args, **kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_generated\operations\_container_operations.py:1063: in get_properties
    pipeline_response: PipelineResponse = self._client._pipeline.run(  # pylint: disable=protected-access
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:229: in run
    return first_node.send(pipeline_request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_redirect.py:197: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_shared\policies.py:556: in send
    raise err
D:\Anaconda3\envs\sr\lib\site-packages\azure\storage\blob\_shared\policies.py:528: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\_base.py:86: in send
    response = self.next.send(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_authentication.py:145: in send
    self.on_request(request)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_authentication.py:120: in on_request
    self._request_token(*self._scopes)
D:\Anaconda3\envs\sr\lib\site-packages\azure\core\pipeline\policies\_authentication.py:94: in _request_token
    self._token = cast(SupportsTokenInfo, self._credential).get_token_info(*scopes, options=options)
D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_internal\decorators.py:23: in wrapper
    token = fn(*args, **kwargs)
D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:125: in get_token_info
    return self._get_token_base(*scopes, options=options)
D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:147: in _get_token_base
    output = _run_command(command, self._process_timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

command = 'az account get-access-token --output json --resource https://storage.azure.com'
timeout = 10

    def _run_command(command: str, timeout: int) -> str:
        # Ensure executable exists in PATH first. This avoids a subprocess call that would fail anyway.
        if shutil.which(EXECUTABLE_NAME) is None:
            raise CredentialUnavailableError(message=CLI_NOT_FOUND)

        if sys.platform.startswith("win"):
            args = ["cmd", "/c", command]
        else:
            args = ["/bin/sh", "-c", command]
        try:
            working_directory = get_safe_working_dir()

            kwargs: Dict[str, Any] = {
                "stderr": subprocess.PIPE,
                "stdin": subprocess.DEVNULL,
                "cwd": working_directory,
                "universal_newlines": True,
                "timeout": timeout,
                "env": dict(os.environ, AZURE_CORE_NO_COLOR="true"),
            }
            return subprocess.check_output(args, **kwargs)
        except subprocess.CalledProcessError as ex:
            # non-zero return from shell
            # Fallback check in case the executable is not found while executing subprocess.
            if ex.returncode == 127 or ex.stderr.startswith("'az' is not recognized"):
>               raise CredentialUnavailableError(message=CLI_NOT_FOUND) from ex
E               azure.identity._exceptions.CredentialUnavailableError: Azure CLI not found on path

D:\Anaconda3\envs\sr\lib\site-packages\azure\identity\_credentials\azure_cli.py:239: CredentialUnavailableError
------------------------------ Captured log call ------------------------------
WARNING  azure.identity._internal.decorators:decorators.py:43 AzureCliCredential.get_token_info failed: Azure CLI not found on path
WARNING  azure.identity._internal.decorators:decorators.py:43 AzureCliCredential.get_token_info failed: Azure CLI not found on path
WARNING  azure.identity._internal.decorators:decorators.py:43 AzureCliCredential.get_token_info failed: Azure CLI not found on path
WARNING  azure.identity._internal.decorators:decorators.py:43 AzureCliCredential.get_token_info failed: Azure CLI not found on path
=========================== short test summary info ===========================
FAILED test_azure.py::test_azure - azure.identity._exceptions.CredentialUnava...
======================== 1 failed in 88.11s (0:01:28) =========================
pvaneck commented 5 days ago

If you were to have the following code in a separate python script, then use the "Run and Debug" feature in VSCode, does it yield the same error?

import subprocess
import os

print(os.environ['PATH'])

args = ["cmd", "/c", "az account --help"]
kwargs = {
    "universal_newlines": True,
    "env": dict(os.environ, AZURE_CORE_NO_COLOR="true"),
}
try:
    output = subprocess.check_output(args, **kwargs)
    print(output)
except subprocess.CalledProcessError as e:
    print(e)

If so, does removing the "cmd" and "/c" entries in the args list help at all? Would also be curious to see how the output from print(os.environ['PATH']) differs at runtime.

I'm not super familiar with all the VS Code mechanisms like the Debug Console vs Integrated Terminal, but seems like creating/updating a launch.json file could also be something to look into. Perhaps something like this might work:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python Debugger: Current File",
            "type": "debugpy",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "purpose": [ "debug-test" ],
            "env": {
                "PATH": "${env:PATH};C:\\path\\to\\azure\\cli"
            }
        }
    ]
}
github-actions[bot] commented 2 days ago

Hi @lovettchris. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

lovettchris commented 1 hour ago

Your sample code above works fine in the debugger - the cmd runs with the correct path and az command completes. I used "az account show" to be sure it fetches something from azure that requires authentication. --help is not enough.

I also tested this in the debugger and it runs fine:

import os
import subprocess

def find_az_command():
    path = os.environ['PATH']
    for p in path.split(os.pathsep):
        if os.path.exists(os.path.join(p, 'az.cmd')):
            return os.path.join(p, 'az.cmd')
        if os.path.exists(os.path.join(p, 'az')):
            return os.path.join(p, 'az')
    print("ERROR: az command not found in PATH")
    return None

def get_account_details():
    az = find_az_command()
    args = [az, "account", "show"]
    kwargs = {
        "universal_newlines": True,
        "env": dict(os.environ, AZURE_CORE_NO_COLOR="true"),
    }
    try:
        output = subprocess.check_output(args, **kwargs)
        return output
    except Exception as e:
        return str(e)

def test_account_details():
    details = get_account_details()
    print(details)

And debugging this as a pytest in the debugger also succeeds, so not sure why my original test_azure function fails in the debugger during pytest.