Could not deploy chalice

[apfromiit]

Logs:

MacBook% chalice deploy

Creating deployment package.
Updating policy for IAM role: ***-server-dev
Updating lambda function: ***-server-dev
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/chalice/awsclient.py", line 904, in _update_function_code
    result = lambda_client.update_function_code(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterValueException: An error occurred (InvalidParameterValueException) when calling the UpdateFunctionCode operation: Unzipped size must be smaller than 117044649 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/chalice/deploy/deployer.py", line 376, in deploy
    return self._deploy(config, chalice_stage_name)
  File "/usr/local/lib/python3.9/site-packages/chalice/deploy/deployer.py", line 392, in _deploy
    self._executor.execute(plan)
  File "/usr/local/lib/python3.9/site-packages/chalice/deploy/executor.py", line 42, in execute
    getattr(self, '_do_%s' % instruction.__class__.__name__.lower(),
  File "/usr/local/lib/python3.9/site-packages/chalice/deploy/executor.py", line 55, in _do_apicall
    result = method(**final_kwargs)
  File "/usr/local/lib/python3.9/site-packages/chalice/awsclient.py", line 882, in update_function
    return_value = self._update_function_code(function_name=function_name,
  File "/usr/local/lib/python3.9/site-packages/chalice/awsclient.py", line 912, in _update_function_code
    raise self._get_lambda_code_deployment_error(e, context)
chalice.awsclient.DeploymentPackageTooLargeError: An error occurred (InvalidParameterValueException) when calling the UpdateFunctionCode operation: Unzipped size must be smaller than 117044649 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/chalice/cli/__init__.py", line 636, in main
    return cli(obj={})
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/chalice/cli/__init__.py", line 189, in deploy
    deployed_values = d.deploy(config, chalice_stage_name=stage)
  File "/usr/local/lib/python3.9/site-packages/chalice/deploy/deployer.py", line 378, in deploy
    raise ChaliceDeploymentError(e)
chalice.deploy.deployer.ChaliceDeploymentError: ERROR - While sending your chalice handler code to Lambda to update function 
"***-server-dev", received the following error:

 An error occurred (InvalidParameterValueException) when calling the 
 UpdateFunctionCode operation: Unzipped size must be smaller than 117044649 
 bytes

To avoid this error, decrease the size of your chalice application by removing 
code or removing dependencies from your chalice application.

MacBook% chalice --version
chalice 1.27.3, python 3.9.8, darwin 21.6.0

Last successfully deployed lambda function properties from AWS:

{
    "FunctionName": "***-server-dev",
    "FunctionArn": "arn:aws:lambda:us-west-2:***:function:***-server-dev:$LATEST",
    "Runtime": "python3.8",
    "Role": "arn:aws:iam::***:role/***-server-dev",
    "Handler": "app.app",
    "CodeSize": 24515982,
    "Description": "",
    "Timeout": 30,
    "MemorySize": 1024,
    "LastModified": "2022-08-05T10:46:12.000+0000",
    "CodeSha256": "HNYq5jI6nepN089U3PYwf0S/0kLGsjvVl50zf5PUjwc=",
    "Version": "$LATEST",
    "VpcConfig": {
        "SubnetIds": [],
        "SecurityGroupIds": [],
        "VpcId": ""
    },
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "2067dba5-7dc1-440e-86e8-9bf8805cc5f2",
    "Layers": [
        {
            "Arn": "arn:aws:lambda:us-west-2:***:layer:mysql-connector-python:1",
            "CodeSize": 6543020
        },
        {
            "Arn": "arn:aws:lambda:us-west-2:770693421928:layer:Klayers-p38-pandas:6",
            "CodeSize": 38553819
        },
        {
            "Arn": "arn:aws:lambda:us-west-2:770693421928:layer:Klayers-p38-beautifulsoup4:1",
            "CodeSize": 366592
        }
    ],
    "PackageType": "Zip",
    "Architectures": [
        "x86_64"
    ]
}

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html Above documentation defines lambda package unzipped size limit as 250MB

But deployment failure logs mentions "Unzipped size must be smaller than 117044649 bytes"

My function code size is 23.38MB with 3 lambda layers of 6.23MB, 36.77MB and 0.35MB

I used klayers as lambda layers.

[apfromiit]

No code changes were made. Deployment was successful 28 days ago (05-Aug-2022). But failing while deploying today.

[apfromiit]

chalice package command creates deployment package of below sizes: Zipped: 2,57,25,387 bytes (24.53MB) Unzipped: 12,12,22,712 bytes (115.60MB)

[apfromiit]

MacBook% cat requirements.txt PyJWT paramiko ansi2html requests pytz Jinja2 twilio markdown firebase-admin jmespath

[apfromiit]

Issue is reproducible with this sample repo.

https://github.com/apfromiit/chalice-deployment-issue

Clone the repo and run chalice deploy

[jamesls]

Looking into this, not sure why it would suddenly start failing if you haven't changed any packages. The error message about the package size is from the Lambda service itself. I'll see what I can find.

[jamesls]

I'm able to deploy your sample repo without any issues:

(tmp-519657db543e5a7) /tmp/chalice-deployment-issue (main *|u=) $ cat requirements.txt
PyJWT
paramiko
ansi2html
requests
pytz
Jinja2
twilio
markdown
firebase-admin
jmespath
mysql-connector-python(tmp-519657db543e5a7) /tmp/chalice-deployment-issue (main *|u=) $ cat .chalice/config.json
{
  "version": "2.0",
  "app_name": "test",
  "stages": {
    "dev": {
      "api_gateway_stage": "api"
    }
  },
  "layers": [
    "arn:aws:lambda:us-west-2:770693421928:layer:Klayers-p38-pandas:7",
    "arn:aws:lambda:us-west-2:770693421928:layer:Klayers-p38-beautifulsoup4:1"
  ]
}

$ chalice deploy
/Users/j/.virtualenvs/tmp-519657db543e5a7/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Creating deployment package.
Reusing existing deployment package.
Creating IAM role: test-dev
Creating lambda function: test-dev
Creating Rest API
Resources deployed:
  - Lambda ARN: arn:aws:lambda:us-west-2:12345:function:test-dev
  - Rest API URL: https://abcd.execute-api.us-west-2.amazonaws.com/api/
(tmp-519657db543e5a7) /tmp/chalice-deployment-issue (main *|u=) $ http https://abcd.execute-api.us-west-2.amazonaws.com/api/
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 17
Content-Type: application/json
Date: Fri, 02 Sep 2022 16:58:52 GMT

{
    "hello": "world"
}

I also tried redeploying the app after making a change and the update case works as well.

There's also a quote for the total amount of storage across all your uploaded functions of 75GB (same doc page you linked). Could that be what you're running into?

Storage for uploaded functions (.zip file archives) and layers. Each function version and layer version consumes storage.

For best practices on managing your code storage, see Monitoring Lambda code storage in the Lambda Operator Guide.

Otherwise I don't know if there's anything we're doing in Chalice could help here, the packaging code hasn't been touched for a while.

[apfromiit]

Added a github workflow to reproduce the issue.

Please refer the deployment failure logs in https://github.com/apfromiit/chalice-deployment-issue/runs/8167791255?check_suite_focus=true

Workflow: https://github.com/apfromiit/chalice-deployment-issue/blob/main/.github/workflows/chalice-deploy.yml

[apfromiit]

The aws credentials provided for above workflow is of new aws account containing only one other lambda function and zero lambda layers. The screenshot is from aws console lambda dashboard.

[jamesls]

I tried the same steps as the github workflow you linked from a new AWS account and I am still able to deploy the application with no errors. Also tried with python3.8 and python3.9 and both cases worked.

One thing that might affects things is that you have no version constraints in your requirements.txt file so it's possible that the latest deployment picked up a newer version of a package that brought the file size over the limit.

The error message from the github workflow says the unzipped package must be smaller than 138217017 which matches what I calculated as what's left after the two layer sizes (unzipped they're about 129MB total, which would leave you with around 256 - 129 or 127MB for your deployment package).

I think either way, you'll have to reduce the deployment package size somehow, I don't think there's anything we can do on Chalice's side, as that error message of Unzipped size must be smaller than 138217017 bytes is coming directly from the Lambda service.

[apfromiit]

This is exactly the cause. Removing pandas layer to allow room for deployment package. Thanks @jamesls for the analysis.

[apfromiit]

Workflow is successful after removing pandas layer. https://github.com/apfromiit/chalice-deployment-issue/runs/8225937572?check_suite_focus=true