az cli logs sensitive info in --DEBUG mode

[harneetvirk]

Related command

az cli logs sensitive info in --DEBUG mode **Is your feature request related to a problem? Please describe.**

Run "az ml data create --debug --file some_dataset.yaml" with your default workspace and resourcegroup. Check the debug output. It contains something like this:

azure.core.pipeline.policies._universal: { "secretsType": "AccountKey", "key": "2ww2c/dCC42vx********" }

Describe the solution you'd like

No secrets are getting exposed from any of the az commands when executing in DEBUG mode.

[yonzhan]

@jiasli for awareness

[jiasli]

I am not aware of az data. Are you developing your own extension?

---

Also, what version of Azure CLI you are using? The HTTP log should now be logged by cli.azure.cli.core.sdk.policies, instead of azure.core.pipeline.policies._universal.

---

As for sensitive information shown in debug log, I think this is a common issue.

az storage commands once face the same issue. https://github.com/Azure/azure-cli/pull/10473 disabled HTTP trace for Track 1 SDK:

https://github.com/Azure/azure-cli/blob/93bbc71faccbef32fe3df3fc5cc953fa1277208d/src/azure-cli/azure/cli/command_modules/storage/_client_factory.py#L144-L147

but client.config.enable_http_logger = False was removed during Track 2 migration (#15845)

https://github.com/Azure/azure-cli/blob/b40afb9cdde133f135fc2c4332b94c9f159734ff/src/azure-cli/azure/cli/command_modules/storage/_client_factory.py#L148-L150

making keys appear again in the debug log:

> az storage account keys list -n myst --debug
...
cli.azure.cli.core.sdk.policies: {"keys":[{"creationTime":"2021-09-24T05:54:33.0930905Z","keyName":"key1","value":"HzdZdX6...","permissions":"FULL"},{"creationTime":"2021-09-24T05:54:33.0930905Z","keyName":"key2","value":"Ht1KOxV...","permissions":"FULL"}]}

Other commands such as az redis list-keys also log keys in the debug log.

[harneetvirk]

Sorry, there was a typo in the description of the issue. This is not a new extension, but we are using ML extension.

az ml data create --name mytestdata01 --file ./src/azure-ai-ml/tests/test_configs/dataset/data_local_path.yaml --version 1 --debug

azure.core.pipeline.policies._universal: Request URL: 'https://management.azure.com/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.MachineLearningServices/workspaces/REDACTED/datastores/workspaceblobstore/listSecrets?api-version=2022-05-01' Request method: 'POST' Request headers: 'Accept': 'application/json' 'x-ms-client-request-id': 'REDACTED' 'User-Agent': 'azureml-cli-v2/0.0.139 azure-ai-ml/0.0.139 azsdk-python-mgmt-machinelearningservices/0.1.0 Python/3.7.10 (Windows-10-10.0.22621-SP0)' 'Authorization': 'Bearer REDACTED' Request body: This request has no body urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.MachineLearningServices/workspaces/REDACTED/datastores/workspaceblobstore/listSecrets?api-version=2022-05-01 HTTP/1.1" 200 None azure.core.pipeline.policies._universal: Response status: '200' Response headers: 'Cache-Control': 'no-cache' 'Pragma': 'no-cache' 'Transfer-Encoding': 'chunked' 'Content-Type': 'application/json; charset=utf-8' 'Content-Encoding': 'gzip' 'Expires': '-1' 'Vary': 'Accept-Encoding' 'x-ms-ratelimit-remaining-subscription-writes': '1199' 'Request-Context': 'appId=cid-v1:REDACTED' 'x-ms-response-type': 'standard' 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' 'X-Content-Type-Options': 'nosniff' 'x-aml-cluster': 'vienna-eastus-01' 'x-request-time': '0.242' 'Server-Timing': 'traceparent;desc="00-REDACTED"' 'x-ms-request-id': 'c070a7b6-e748-452a-8083-930ce858f8f7' 'x-ms-correlation-request-id': 'REDACTED' 'x-ms-routing-request-id': 'WESTUS:20220926T162712Z:REDACTED' 'Date': 'Mon, 26 Sep 2022 16:27:11 GMT' Response content: { "secretsType": "AccountKey", "key": "REDACTEDKEY==" }

[jiasli]

--debug logging sensitive information is by design, as the very purpose of --debug is to print the raw HTTP trace. To prevent unexpected credential leak, please do not use --debug for sensitive commands.

If you can provide strong business justification that we should make --debug mask secrets and introduce additional switch such as --debug-with-secrets for the original behavior, we are very willing to discuss with our PMs about it.

[BalassaMarton]

I'd recommend adding a --debug-no-secrets switch. The business justification is that we will be submitting bug reports for Azure CLI and the issue template requires us to paste full debug output.

[evelyn-ys]

similar ask https://github.com/Azure/azure-cli/issues/29409