search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4.03k stars 3k forks source link

az rest fails with encoding error when sending UTF-8 emoji characters #30366

Open nimro opened 2 hours ago

nimro commented 2 hours ago

Describe the bug

When sending UTF-8 encoded text containing emoji characters (e.g. ✅ \u2705 or 💩 \U0001f4a9), az rest fails with a character encoding error.

Related command

az rest

Errors

The command failed with an unexpected error. Here is the traceback:
'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/util/custom.py", line 24, in rest_call
    r = send_raw_request(cmd.cli_ctx, method, url, headers, uri_parameters, body,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 1004, in send_raw_request
    r = s.send(prepped, **settings)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/az/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/az/lib/python3.11/http/client.py", line 1298, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/az/lib/python3.11/http/client.py", line 1343, in _send_request
    body = _encode(body, 'body')
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/http/client.py", line 166, in _encode
    raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

Issue script & Debug output

Command: az rest --method post --uri https://<servicebus url>/<topic name>/messages --body '✅' --resource https://servicebus.azure.net Debug output:

cli.knack.cli: Command arguments: ['rest', '--method', 'post', '--uri', '[redacted]', '--body', '✅', '--resource', 'https://servicebus.azure.net', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7efe0d68a660>, <function OutputProducer.on_global_arguments at 0x7efe0d3ddb20>, <function CLIQuery.on_global_arguments at 0x7efe0d417600>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'rest': ['azure.cli.command_modules.util']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: util                      0.001         3         7
cli.azure.cli.core: Total (1)                 0.001         3         7
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 3 groups, 7 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : rest
cli.azure.cli.core: Command table: rest
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7efe0c659e40>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/duncanblair/.azure/commands/2024-11-15.12-40-39.rest.1830.log'.
az_command_data_logger: command args: rest --method {} --uri {} --body {} --resource {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7efe0c6905e0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7efe0c44a520>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7efe0c44a660>, <function register_upcoming_breaking_change_info.<locals>.update_breaking_change_info at 0x7efe0c44a700>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7efe0d3ddbc0>, <function CLIQuery.handle_query_parameter at 0x7efe0d4176a0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7efe0c44a5c0>]
cli.azure.cli.core.util: invalid character '✅' (U+2705) (<unknown>, line 1)
cli.azure.cli.core.util: Retrieving token for resource https://servicebus.azure.net
cli.azure.cli.core.auth.persistence: build_persistence: location='/home/duncanblair/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /home/duncanblair/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/[redacted]
msal.authority: openid_config("https://login.microsoftonline.com/[redacted]/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/[redacted]/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/[redacted]/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/[redacted]/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/[redacted]/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/[redacted]/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/[redacted]/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/[redacted]/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core.auth.msal_credentials: UserCredential.get_token: scopes=('https://servicebus.azure.net/.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 7b32327b-7a5a-4994-9b1f-4416ede5eabc
cli.azure.cli.core.util: Request URL: '[redacted]'
cli.azure.cli.core.util: Request method: 'POST'
cli.azure.cli.core.util: Request headers:
cli.azure.cli.core.util:     'User-Agent': 'python/3.11.8 (Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39) AZURECLI/2.65.0 (DEB)'
cli.azure.cli.core.util:     'Accept-Encoding': 'gzip, deflate'
cli.azure.cli.core.util:     'Accept': '*/*'
cli.azure.cli.core.util:     'Connection': 'keep-alive'
cli.azure.cli.core.util:     'x-ms-client-request-id': 'fcde9c1c-c539-442d-9f60-a2811ab8a2ae'
cli.azure.cli.core.util:     'CommandName': 'rest'
cli.azure.cli.core.util:     'ParameterSetName': '--method --uri --body --resource --debug'
cli.azure.cli.core.util:     'Authorization': 'Bearer [redacted]'
cli.azure.cli.core.util:     'Content-Length': '3'
cli.azure.cli.core.util: Request body:
cli.azure.cli.core.util: ✅
urllib3.connectionpool: Starting new HTTPS connection (1): sb-internal-dev.servicebus.windows.net:443
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/util/custom.py", line 24, in rest_call
    r = send_raw_request(cmd.cli_ctx, method, url, headers, uri_parameters, body,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 1004, in send_raw_request
    r = s.send(prepped, **settings)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/az/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/az/lib/python3.11/http/client.py", line 1298, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/az/lib/python3.11/http/client.py", line 1343, in _send_request
    body = _encode(body, 'body')
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/http/client.py", line 166, in _encode
    raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.

cli.azure.cli.core.azclierror: The command failed with an unexpected error. Here is the traceback:
az_command_data_logger: The command failed with an unexpected error. Here is the traceback:
cli.azure.cli.core.azclierror: 'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/util/custom.py", line 24, in rest_call
    r = send_raw_request(cmd.cli_ctx, method, url, headers, uri_parameters, body,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 1004, in send_raw_request
    r = s.send(prepped, **settings)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/az/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/az/lib/python3.11/http/client.py", line 1298, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/az/lib/python3.11/http/client.py", line 1343, in _send_request
    body = _encode(body, 'body')
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/http/client.py", line 166, in _encode
    raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
az_command_data_logger: 'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 666, in execute
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 703, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 336, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/util/custom.py", line 24, in rest_call
    r = send_raw_request(cmd.cli_ctx, method, url, headers, uri_parameters, body,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 1004, in send_raw_request
    r = s.send(prepped, **settings)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/az/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/opt/az/lib/python3.11/http/client.py", line 1298, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/az/lib/python3.11/http/client.py", line 1343, in _send_request
    body = _encode(body, 'body')
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/http/client.py", line 166, in _encode
    raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2705' in position 0: Body ('✅') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7efe0c65a0c0>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.361 seconds (init: 0.105, invoke: 0.256)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 8078 in cache file under /home/duncanblair/.azure/telemetry/20241115124039901
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /home/duncanblair/.azure /home/duncanblair/.azure/telemetry/20241115124039901"
telemetry.process: Return from creating process 1848
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The request is sent successfully.

Environment Summary

azure-cli 2.65.0 *

core 2.65.0 * telemetry 1.1.0

Extensions: azure-devops 0.25.0 containerapp 0.3.41 front-door 1.0.17 serviceconnector-passwordless 0.3.11 storage-preview 0.8.3

Dependencies: msal 1.31.0 azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3' Extensions directory '/home/duncanblair/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Sep 25 2024, 11:33:44) [GCC 11.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

azure-client-tools-bot-prd[bot] commented 2 hours ago

Hi @nimro,

2.65.0 is not the latest Azure CLI(2.66.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

yonzhan commented 30 minutes ago

az rest