Executor Invocation Error due to permissions denied

[regoawt]

Hi,

I have a Python Lambda function fronted with a function URL. I am able to invoke it without issue using curl or python requests.

Expected behaviour

Power tuning tool runs without error

Actual behaviour

When I try and run it through the power tuning tool, it always fails with an Invocation Error:

{
  "cause": {
    "errorType": "Error",
    "trace": [
      "Error: Invocation error (running in series): 123,34,101,114,114,111,114,77,101,115,115,97,103,101,34,58,32,34,91,69,114,114,110,111,32,49,51,93,32,80,101,114,109,105,115,115,105,111,110,32,100,101,110,105,101,100,58,32,39,47,118,97,114,47,116,97,115,107,47,109,97,105,110,46,112,121,39,34,44,32,34,101,11... with payload <redacted>",
      "    at runInSeries (/var/task/executor.js:167:19)",
      "    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
      "    at async module.exports.handler (/var/task/executor.js:65:19)"
    ]
  },
  "error": "Error"
}

Note I've removed the errorMessage item and truncated the list of numbers in the trace for brevity.

Looking at the actual invocation logs of my Lambda function, I am getting a PermissionError:

{
    "timestamp": "2024-05-18T13:13:38Z",
    "log_level": "ERROR",
    "errorMessage": "[Errno 13] Permission denied: '/var/task/main.py'",
    "errorType": "PermissionError",
    "requestId": "",
    "stackTrace": [
        "  File \"/var/lang/lib/python3.12/importlib/__init__.py\", line 90, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n",
        "  File \"<frozen importlib._bootstrap>\", line 1387, in _gcd_import\n",
        "  File \"<frozen importlib._bootstrap>\", line 1360, in _find_and_load\n",
        "  File \"<frozen importlib._bootstrap>\", line 1331, in _find_and_load_unlocked\n",
        "  File \"<frozen importlib._bootstrap>\", line 935, in _load_unlocked\n",
        "  File \"<frozen importlib._bootstrap_external>\", line 991, in exec_module\n",
        "  File \"<frozen importlib._bootstrap_external>\", line 1128, in get_code\n",
        "  File \"<frozen importlib._bootstrap_external>\", line 1186, in get_data\n"
    ]
}

Fixes attempted

I have tried to modify the file/directory permissions using chmod but that did not resolve anything, as described in this SO issue.

Environment

• MacOS
• The function is bundled, zipped and deployed with Python CDK.

[alexcasalboni]

Hey @regoawt thanks for reaching out 🙏

This looks very similar to #242 and it could be related to the function timing out.

247 has implemented a way to log the exact function invocation error (rather than this stream/array of numbers) and it's been merged, although it hasn't been published as a new SAR version.

I imagine you're deploying using SAR via CDK, right?

I'd suggest to deploy the latest Lambda Power Tuning without using SAR and check if you get a better invocation error.

[regoawt]

@alexcasalboni thanks for the quick reply!

Will take a look at the linked issue. Yes I have deployed using SAR - will deploy the latest version using other means.

[regoawt]

Managed to resolve the issue.

I still don't have a good explanation for why I was only getting the error when running the function through the Power Tuner, but as shown in the function error log, it was to do with the permissions of the Lambda package files. I found this issue: https://github.com/aws/aws-cdk/issues/8707#issuecomment-649310687 which pointed me in the right direction.

I had to configure the CDK bundling options of my function to give full permission to every file in the package using chmod:

code=_lambda.Code.from_asset(
    path=os.path.join(os.path.dirname(__file__), "src"),
    bundling=BundlingOptions(
        image=_lambda.Runtime.PYTHON_3_12.bundling_image,
        command=[
            "bash",
            "-c",
            "&&".join(
                [
                    "pip install -r requirements.txt --no-deps --platform manylinux2014_x86_64 -t /asset-output",
                    "cp -au . /asset-output",
                    "chmod -R 777 /asset-output",
                ]
            ),
        ],
        platform="linux/amd64",
    ),
),

[regoawt]

PS it works with 755 permissions as per https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-deployment.html#troubleshooting-deployment-denied