on MacOS, `az functionapp deployment source config-zip` causes `Cannot allocate memory` often

s-nakagaki commented 1 month ago

Describe the bug

Trying to deploy a windows function via azure-cli on VSCode zsh terminal of macOS. Cannot allocate memory happens so many times. (NOT always)

According to --debug result, polling requests causes memory allocation error.

My Mac has 24 GB memory and there are 5GB free memory at least. Of course, dotnet build is succeeded at local.

There is no error in the log of deploy center of azure function.

Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Creating directory "C:\local\Temp\zipdeploy\extracted\...
Copying file from "C:\home\.nuget\...
Creating directory "C:\local\Temp\zipdeploy\extracted\...
Copying file from "C:\home\.nuget\...
Thread was being aborted.
Copying file from "C:\home\.nuget\...
Thread was being aborted.

Related command

az functionapp deployment source config-zip \ -g ${RESOURCE_GROUP} \ -n ${FUNCTION_NAME} \ --build-remote true \ --src ./.zip/${FUNCTION_NAME}.zip \ --timeout 1200

Errors

Getting scm site credentials for zip deployment Starting zip deployment. This operation can take a while to complete ... Deployment endpoint responded with status code 202 The command failed with an unexpected error. Here is the traceback: HTTPSConnectionPool(host='func-hoge.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/deployments/latest (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory')) Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) OSError: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request self._validate_conn(conn) File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn conn.connect() File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() ^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/adapters.py", line 564, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='func-hoge.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/deployments/latest (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 664, in execute raise ex File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 731, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 701, in _run_job result = cmd_copy(params) ^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 334, in call return self.handler(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler return op(command_args) ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/command_modules/appservice/custom.py", line 677, in enable_zip_deploy_functionapp return enable_zip_deploy(cmd, resource_group_name, name, src, timeout, slot) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/command_modules/appservice/custom.py", line 804, in enable_zip_deploy response_body = _check_zip_deployment_status(cmd, resource_group_name, name, deployment_status_url, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/command_modules/appservice/custom.py", line 5628, in _check_zip_deployment_status response = requests.get(deployment_status_url, headers=headers, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/adapters.py", line 597, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='func-hoge.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/deployments/latest (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory'))

Issue script & Debug output

urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585 cli.azure.cli.command_modules.appservice.custom: Running deployment command... urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443 cli.azure.cli.core.azclierror: Traceback (most recent call last): File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection(

Expected behavior

no Cannot allocate memory

Environment Summary

azure-cli 2.62.0

core 2.62.0 telemetry 1.1.0

Extensions: account 0.2.5

Dependencies: msal 1.28.1 azure-mgmt-resource 23.1.1

Python location '/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/bin/python' Extensions directory '/Users/hoge/.azure/cliextensions'

Python (Darwin) 3.11.9 (main, Apr 2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

yonzhan commented 1 month ago

Thank you for opening this issue, we will look into it.

github-actions[bot] commented 1 month ago

Here are some similar issues that might help you. Please check if they can solve your problem.

23593

microsoft-github-policy-service[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kamperiadis, @khkh-ms, @shreyabatra4.

s-nakagaki commented 1 month ago

~~This might happen only on VSCode zsh terminal. I'm not sure.~~ This happens on macOS native terminal too.

s-nakagaki commented 1 month ago

When this happened, log view of deploy center of azure function in azure portal said Too many requests. Please try again later..

I guess, the retrying sequence of azure cli might not wait for response and might retry to connect very soon without cleaning-up old connection. So at client side, there are so many memory allocation for connection, at server side, there are so many request to check the deploy status.

s-nakagaki commented 1 month ago

the deploy is failing over and over. This is so stressful.

s-nakagaki commented 1 month ago

After re-creating Function, service plan and storage, the deploy succeeded finally. There was no difference of code with before and after.

Azure / azure-cli