Open nataizya opened 2 months ago
Thanks for reporting this issue.
Can you try changing your runtime to provided.al2023
to see if it works? I think there was a change in the rust builder we used under the hood, they changed default runtime settings to be aligned with newer runtimes, which broke the provided.al2
ones.
We've merged a fix to align with the change above, however it is not been released yet. If you confirm that changing to newer runtime works, we can assume that new SAM CLI release should fix the issue.
Thanks for responding @mndeveci.
I did try it with provided.al2023
as well and faced the same issue.
Logs:
2024-05-13 23:44:07,411 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml
2024-05-13 23:44:07,415 | Loading configuration values from [default.['local', 'invoke'].parameters]
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml'...
2024-05-13 23:44:07,416 | Configuration values successfully loaded.
2024-05-13 23:44:07,416 | Configuration values are: {'stack_name': 'sam-app-2'}
2024-05-13 23:44:07,420 | Using SAM Template at
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
2024-05-13 23:44:07,480 | Using config file: samconfig.toml, config environment: default
2024-05-13 23:44:07,481 | Expand command line arguments to:
2024-05-13 23:44:07,481 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost
--container_host_interface=127.0.0.1
2024-05-13 23:44:07,482 | local invoke command is called
2024-05-13 23:44:07,486 | No Parameters detected in the template
2024-05-13 23:44:07,505 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:44:07,506 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:44:07,507 | 0 stacks found in the template
2024-05-13 23:44:07,508 | No Parameters detected in the template
2024-05-13 23:44:07,524 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:44:07,524 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:44:07,525 | 2 resources found in the stack
2024-05-13 23:44:07,526 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-05-13 23:44:07,526 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
2024-05-13 23:44:07,539 | Found one Lambda function with name 'HelloWorldFunction'
2024-05-13 23:44:07,540 | Invoking bootstrap (provided.al2023)
2024-05-13 23:44:07,540 | No environment variables found for function 'HelloWorldFunction'
2024-05-13 23:44:07,541 | Loading AWS credentials from session with profile 'None'
2024-05-13 23:44:07,548 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,549 | Resolved absolute path to code is
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,550 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,550 | Resolved real code path to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
2024-05-13 23:44:07,551 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction is
not a zip/jar file
2024-05-13 23:44:13,273 | Local image is up-to-date
2024-05-13 23:44:13,292 | Checking free port on 127.0.0.1:8347
2024-05-13 23:44:13,299 | Using local image: public.ecr.aws/lambda/provided:al2023-rapid-x86_64.
2024-05-13 23:44:13,301 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction
as /var/task:ro,delegated, inside runtime container
2024-05-13 23:44:13,698 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2024-05-13 23:44:13,701 | Getting lock for the key localhost-8347
2024-05-13 23:44:13,702 | Waiting to retrieve the lock (localhost-8347) to start invocation
START RequestId: 55fe8a55-ca1d-436c-a3ed-eec2c3fda782 Version: $LATEST
13 May 2024 21:44:13,759 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:44:13,762 [ERROR] (rapid) Invoke failed InvokeID=261e53eb-7616-45bf-96fc-b24359daf3d3 error=Runtime exited with error: signal: illegal instruction
13 May 2024 21:44:13,763 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:44:13,767 | Failed to deserialize response from RIE, returning the raw response as is
2024-05-13 23:44:14,790 | [Container state] OOMKilled False
2024-05-13 23:44:14,925 | Cleaning all decompressed code dirs
2024-05-13 23:44:14,927 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:44:15,027 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:44:15,028 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'dc3cb305-75d5-4d40-b3b7-d281511cda09',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '9e7669abfc305db5598caf057dd1af7044b0710b8c9f99f979d86cc7bb8b69ca', 'initialCommit': None},
'duration': 7446, 'exitReason': 'success', 'exitCode': 0}}]}
2024-05-13 23:44:15,028 | Unable to find Click Context for getting session_id.
2024-05-13 23:44:15,032 | Sending Telemetry: {'metrics': [{'events': {'requestId': '48b8995e-d5d3-43ca-9aba-def06169a914',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'4a072847aba84d9eae8720975249e7d7', 'time_stamp': '2024-05-13 21:44:07.411', 'exception_name': None}, {'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'd0e7eb37fe8247958066a2e7d2574cd1', 'time_stamp': '2024-05-13
21:44:07.480', 'exception_name': None}]}}}]}
2024-05-13 23:44:15,977 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:44:15,978 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:44:15,986 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:44:15,988 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':
'd840b3ea-0bfe-4bde-8fe8-33eaee4876d5', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':
'699a40ef-6b4d-474d-8610-c8a5640b79f5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':
'1.116.0', 'runtimes': ['provided.al2023']}}]}
2024-05-13 23:44:16,792 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
nataizya@Nataizyas-MacBook-Air sam-app-2 % cd ../sam-app
nataizya@Nataizyas-MacBook-Air sam-app % sam local invoke --debug
2024-05-13 23:46:14,648 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml
2024-05-13 23:46:14,651 | Loading configuration values from [default.['local', 'invoke'].parameters]
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml'...
2024-05-13 23:46:14,652 | Configuration values successfully loaded.
2024-05-13 23:46:14,652 | Configuration values are: {'stack_name': 'sam-app'}
2024-05-13 23:46:14,656 | Using SAM Template at
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
2024-05-13 23:46:14,702 | Using config file: samconfig.toml, config environment: default
2024-05-13 23:46:14,703 | Expand command line arguments to:
2024-05-13 23:46:14,704 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost
--container_host_interface=127.0.0.1
2024-05-13 23:46:14,705 | local invoke command is called
2024-05-13 23:46:14,708 | No Parameters detected in the template
2024-05-13 23:46:14,727 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:46:14,728 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:46:14,729 | 0 stacks found in the template
2024-05-13 23:46:14,730 | No Parameters detected in the template
2024-05-13 23:46:14,746 | Sam customer defined id is more priority than other IDs. Customer defined id for resource
HelloWorldFunction is HelloWorldFunction
2024-05-13 23:46:14,746 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use
the resource logical id as the resource id
2024-05-13 23:46:14,747 | 2 resources found in the stack
2024-05-13 23:46:14,748 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2024-05-13 23:46:14,748 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml
2024-05-13 23:46:14,779 | Found one Lambda function with name 'HelloWorldFunction'
2024-05-13 23:46:14,779 | Invoking bootstrap (provided.al2)
2024-05-13 23:46:14,780 | No environment variables found for function 'HelloWorldFunction'
2024-05-13 23:46:14,780 | Loading AWS credentials from session with profile 'None'
2024-05-13 23:46:14,788 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,789 | Resolved absolute path to code is
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,790 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,790 | Resolved real code path to
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction
2024-05-13 23:46:14,791 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction is not
a zip/jar file
2024-05-13 23:46:20,664 | Local image is up-to-date
2024-05-13 23:46:20,685 | Checking free port on 127.0.0.1:5726
2024-05-13 23:46:20,695 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.
2024-05-13 23:46:20,697 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction as
/var/task:ro,delegated, inside runtime container
2024-05-13 23:46:20,949 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2024-05-13 23:46:20,953 | Getting lock for the key localhost-5726
2024-05-13 23:46:20,955 | Waiting to retrieve the lock (localhost-5726) to start invocation
2024-05-13 23:46:21,068 | Getting lock for the key localhost-5726
2024-05-13 23:46:21,072 | Waiting to retrieve the lock (localhost-5726) to start invocation
START RequestId: 2d02aba6-328d-4a67-b94f-91b4058b407e Version: $LATEST
13 May 2024 21:46:21,114 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:46:21,118 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=3bde647e-a338-4ff7-b860-7e24e6d8ff10
13 May 2024 21:46:21,120 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:46:21,122 | Failed to deserialize response from RIE, returning the raw response as is
2024-05-13 23:46:22,157 | [Container state] OOMKilled False
2024-05-13 23:46:22,288 | Cleaning all decompressed code dirs
2024-05-13 23:46:22,291 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:22,404 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:22,406 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '88d70caf-438d-43c5-bdb9-cd8b910a8459',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None},
'duration': 7588, 'exitReason': 'success', 'exitCode': 0}}]}
2024-05-13 23:46:22,407 | Unable to find Click Context for getting session_id.
2024-05-13 23:46:22,411 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a5e10d43-7cbc-43b6-be11-7dabe9d0a741',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'ba1472d1be8a4258bce10fcb3747afbc', 'time_stamp': '2024-05-13 21:46:14.647', 'exception_name': None}, {'event_name':
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '4d7d2b65326940d7aebc4265646dfc81', 'time_stamp': '2024-05-13
21:46:14.702', 'exception_name': None}]}}}]}
2024-05-13 23:46:23,369 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:46:23,375 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
2024-05-13 23:46:23,379 | Telemetry endpoint configured to be
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-13 23:46:23,381 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':
'a53adc53-679e-478a-9032-a01c08bee7d2', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':
'eab61e43-2617-4a4b-8b64-239884235d30', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':
'1.116.0', 'runtimes': ['provided.al2']}}]}
2024-05-13 23:46:24,182 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read
timed out. (read timeout=0.1)
Thanks for confirmation!
Can you also confirm what happens when you deploy this function? As far as we can see, when this function is deployed it works as expected, however running locally fails with the error that you mentioned above.
From our side we validated following scenarios;
sam build
), locally invoking fails (sam local invoke
)sam build
), locally invoking succeeds (sam local invoke
)sam local invoke
@nataizya we've tried with a Docker alternative, and it worked without issues. I believe this is coming from Docker Desktop's Apple Silicon distribution since it also works on Intel machines.
I suspect that the deployment will work as well though I havent tested that.
Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?
Before going further down the list, you may try to uninstall/remove Docker Desktop completely and install it from their official installers, to see if it is going to fix your problem (I've seen that sometimes a clean installation fixes their issues).
I can't open an issue on their repository since they are asking more details which I can't provide since I am using an Intel Macbook. If you can, I will recommend creating an issue in this repository here; https://github.com/docker/for-mac/issues
Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?
There are alternative tools which can replace Docker Desktop, and they provide similar experience to use containers on your machine. Some of those are podman or rancher.
Local invocation works for me after change to Architectures: - arm64
. I had to remove all the containers before run the invoke command.
Setting the architecture to arm64
allows it to work. Thanks!!!
Please keep in mind that, if you change Architecture
value in the template.yaml
it will also change the architecture of the deployed function. Architecture of the developer's local machine can be different from what they are planning to deploy in their account. In this case, I believe Docker Desktop for Apple Silicon has some issues with executing cross architecture code.
Noted. I will try and make use of podman or rancher. But for now I can at least test the function locally by temporarily switching to arm64
in the template.yaml
.
As another workaround (which might be better since this doesn't require architecture change), you can also build Rust x86 function with SAM_BUILD_MODE=debug sam build
which will build in debug mode, and it looks like it works with sam local invoke
.
!!! Please don't use SAM_BUILD_MODE=debug
when doing a deployment as it removes the --release
flag for Rust compiler. !!!
I think we found the root cause (thanks to @lucashuy).
There was a change in cargo-lambda
package which is used for building Rust functions under the hood. This PR changed the default target cpu (microarch) to be haswell
. It looks like this is causing issues when executable built with that flag. If it is changed to x86_64
then it works fine.
However the reason for the change above was to make it more optimized for the target lambda runtime. So this code works when it is deployed (as intended) however when it is tried to be emulated on Apple Silicon machines, it doesn't work (possible) missing support from Docker Desktop.
We don't want to change default target since this was selected due to optimizations, for now we would ask customers to use workaround to build with SAM_BUILD_MODE=debug sam build
when testing their lambda functions locally.
Thanks @mndeveci .
I can confirm that using SAM_BUILD_MODE=debug sam build
does work for local testing.
Thanks all for working with us on this issue. Just to sum up for future folks popping into this issue: SAM CLI builds Rust Functions for "release mode", meaning that specific architecture level optimizations are applied when building the binary. Unfortunately, these optimizations do not work when invoking them in an x86 container, on an arm64 Mac.
Before building the Lambda function, set the SAM_BUILD_MODE=debug
environment variable before running sam build
, like such: SAM_BUILD_MODE=debug sam build
Please note that this is only recommended when building and testing locally. We advise customers to switch back to a release build by unsetting or removing the SAM_BUILD_MODE
environ variable.
Our team is going to prioritize adding a disclaimer message saying the essentially the same thing inside of sam local
.
Description:
I am trying to setup a rust lambda function and to test, I am running
sam local invoke
. This fails with:The same command works fine on an Intel Mac.
Steps to reproduce:
On an Apple Silicon (e.g. M2) machine, setup a Rust lambda:
sam init
AWS Quick Start Templates
Hello World Example
rust (provided.al2)
N
for the following options.sam build
<--- this succeedssam local invoke
Observed result:
Expected result:
Successfully run the
sam local invoke
command and run the lambda function.Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
MacOS 14.4.1 (23E224)
sam --version
:SAM CLI, version 1.116.0
Add --debug flag to command you are running