Closed markmashworth closed 8 months ago
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.
Thank you for reaching out @markmashworth
Could you try to increase the Lambda environment startup timeout? For example: LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=30
(default is 10 seconds)
To mitigate startup delays, we recommend adding the following volume mount for caching (see Installation):
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
You can also try to pull the latest image docker pull localstack/localstack-pro
because we recently started shipping the lambda runtime init, an asset that was previously downloaded lazily.
The used Lambda images can also be updated through our CLI using localstack update all
.
FYI: We are going to ship another improvement in the coming weeks that will prevent the infinite re-try loop in this case.
I have the same problem when invoke lambda function, here is what I got from lambda execution container:
time="2023-08-26T15:27:01Z" level=debug msg="No code archives set. Skipping download." func=main.DownloadCodeArchives file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/codearchive.go:21"
time="2023-08-26T15:27:01Z" level=debug msg="DNS server disabled." func=main.RunDNSRewriter file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/awsutil.go:143"
time="2023-08-26T15:27:01Z" level=debug msg="Process running as root user." func=main.main file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/main.go:146" euid=0 gid=0 uid=0 username=root
time="2023-08-26T15:27:01Z" level=debug msg="Process running as non-root user." func=main.main file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/main.go:148" euid=993 gid=990 uid=993 username=sbx_user1051
2023-08-26T15:27:01Z [Info] Initializing AWS X-Ray daemon unknown
2023-08-26T15:27:01Z [Debug] Listening on UDP 127.0.0.1:2000
2023-08-26T15:27:01Z [Info] Using buffer memory limit of 78 MB
2023-08-26T15:27:01Z [Info] 1248 segment buffers allocated
2023-08-26T15:27:01Z [Debug] Using Endpoint read from Config file: http://172.17.0.1:4566
2023-08-26T15:27:01Z [Debug] Using proxy address:
2023-08-26T15:27:01Z [Debug] Fetch region us-east-1 from commandline/config file
2023-08-26T15:27:01Z [Info] Using region: us-east-1
2023-08-26T15:27:01Z [Info] HTTP Proxy server using X-Ray Endpoint : http://172.17.0.1:4566
2023-08-26T15:27:01Z [Debug] Using Endpoint: http://172.17.0.1:4566
2023-08-26T15:27:01Z [Debug] Batch size: 10
2023-08-26T15:27:01Z [Info] Starting proxy http server on 127.0.0.1:2000
time="2023-08-26T15:27:01Z" level=debug msg="Hot reloading disabled." func=main.RunHotReloadingListener file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/awsutil.go:160"
time="2023-08-26T15:27:01Z" level=debug msg="Runtime API Server listening on 127.0.0.1:9001" func="go.amzn.com/lambda/rapi.(*Server).Listen" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapi/server.go:94"
time="2023-08-26T15:27:01Z" level=info msg="Configure environment for Init Caching." func="go.amzn.com/lambda/rapid.(*rapidContext).acceptStartRequestForInitCaching" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:391"
time="2023-08-26T15:27:01Z" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory" func=go.amzn.com/lambda/rapid.extensionsDisabledByLayer file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:363"
time="2023-08-26T15:27:01Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory"
time="2023-08-26T15:27:01Z" level=debug msg="Preregister runtime" func=go.amzn.com/lambda/rapid.doInit file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:189"
time="2023-08-26T15:27:01Z" level=debug msg="Start runtime" func=go.amzn.com/lambda/rapid.doInit file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:222"
time="2023-08-26T15:27:01Z" level=debug msg="Received RUNNING" func="go.amzn.com/lambda/rapidcore.(*Server).Init" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapidcore/server.go:533"
Traceback (most recent call last):
File "/var/runtime/bootstrap.py", line 60, in <module>
main()
File "/var/runtime/bootstrap.py", line 57, in main
awslambdaricmain.main([os.environ["LAMBDA_TASK_ROOT"], os.environ["_HANDLER"]])
File "/var/lang/lib/python3.9/os.py", line 679, in __getitem__
raise KeyError(key) from None
KeyError: '_HANDLER'
time="2023-08-26T15:27:01Z" level=warning msg="First fatal error stored in appctx: Runtime.ExitError" func=go.amzn.com/lambda/appctx.StoreFirstFatalError file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/appctx/appctxutil.go:156"
time="2023-08-26T15:27:01Z" level=warning msg="Process 17(bootstrap) exited: Runtime exited with error: exit status 1" func="go.amzn.com/lambda/core.(*Watchdog).GoWait.func1" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/core/watchdog.go:67"
time="2023-08-26T15:27:01Z" level=debug msg="Canceling flows: Runtime exited with error: exit status 1" func="go.amzn.com/lambda/core.(*Watchdog).CancelFlows.func1" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/core/watchdog.go:82"
time="2023-08-26T15:27:01Z" level=error msg="Init failed" func=go.amzn.com/lambda/rapid.handleStartError file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:478" InvokeID= error="Runtime exited with error: exit status 1"
time="2023-08-26T15:27:01Z" level=info msg="blocking the credentials service" func="go.amzn.com/lambda/core.(*credentialsServiceImpl).BlockService" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/core/credentials.go:84"
time="2023-08-26T15:27:01Z" level=debug msg="Dispatching DONE:initCorrelationID" func="go.amzn.com/lambda/rapidcore.(*Server).dispatchDone" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapidcore/server.go:472"
edit: problem solved by adding PROVIDER_OVERRIDE_LAMBDA=legacy
to docker compose file.
Hi @markmashworth and @Sotatek-HaiTrieu2
We just shipped major improvements to the Lambda data plane that fix the behavior where the "same error get logged repeatedly by the container" and provide better insights into errors and timeouts at the Lambda runtime startup (e.g., including container logs). Can you please pull the latest Docker image and try again with debug logs enabled using DEBUG=1
?
The root cause is likely something else but our latest version hopefully helps to debug the issue quicker 📈
@markmashworth Please ensure to build the Numpy layer with the right target architecture in mind when deploying on an Apple Silicon ARM device.
Hi there! Do you know when the changes will be released? We are running into lambda startup errors and big timeouts a lot with 2.2.0 [OSX M1]
FYI my docker version is 2.2.1-dev already and it's not working. Lambda creation is around 20s and lambda invocation doesn't work at all.
Thank you
Thank you for sharing @kirrg001
The changes (i.e., Lambda improvements) are in latest
and expected to be released later this week.
Have you tried the latest
Docker image? What are the errors when running with DEBUG=1
?
Hi @joe4dev We are using the latest image to try to create and invoke a Lambda function. The creation was successful, but the invocation fails. I have attached the complete log for your reference.
Here is my docker-compose.yml:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack:latest
hostname: localstack
ports:
- "4566:4566" # LocalStack Gateway
- "4510-4559:4510-4559" # external services port range
environment:
- DEBUG=1
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
localstack_main | 2023-09-26T12:56:46.937 INFO --- [ asgi_gw_1] localstack.request.aws : AWS lambda.CreateFunction => 201
localstack_main | 2023-09-26T12:56:46.939 DEBUG --- [rvice-task_1] l.s.l.i.lambda_models : Saving code wrapped-async-3cce0d01-da06-4add-b3ce-e4de0fc8bce5 to disk
localstack_main | 2023-09-26T12:56:46.968 DEBUG --- [rvice-task_1] localstack.utils.run : Executing command: ['unzip', '-o', '-q', '/tmp/tmp9aba_w54']
localstack_main | 2023-09-26T12:56:46.969 DEBUG --- [rvice-task_1] l.s.l.i.version_manager : Version preparation of function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async:$LATEST took 30.51ms
localstack_main | 2023-09-26T12:56:46.969 DEBUG --- [rvice-task_1] l.s.l.i.version_manager : Changing Lambda arn:aws:lambda:us-east-2:000000000000:function:wrapped-async:$LATEST (id c73395d7) to active
localstack_main | 2023-09-26T12:56:46.969 DEBUG --- [rvice-task_1] l.s.l.i.event_manager : Starting event manager arn:aws:lambda:us-east-2:000000000000:function:wrapped-async:$LATEST id 281471252265424
localstack_main | 2023-09-26T12:56:46.972 DEBUG --- [ asgi_gw_1] l.services.sqs.provider : creating queue key=wrapped-async-b14816657d83d0168e4769eb22c58dd5 attributes=None tags=None
localstack_main | 2023-09-26T12:56:47.091 INFO --- [ asgi_gw_2] localstack.request.aws : AWS lambda.GetFunctionConfiguration => 200
localstack_main | 2023-09-26T12:56:47.765 DEBUG --- [ asgi_gw_0] l.s.l.i.assignment : Starting new environment
localstack_main | 2023-09-26T12:56:47.765 DEBUG --- [ asgi_gw_0] l.s.l.i.docker_runtime_exe : Creating service endpoint for function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async:$LATEST executor 916ddeabb4d4841522bc81436a4aa00c
localstack_main | 2023-09-26T12:56:47.765 DEBUG --- [ asgi_gw_0] l.s.l.i.docker_runtime_exe : Finished creating service endpoint for function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async:$LATEST executor 916ddeabb4d4841522bc81436a4aa00c
localstack_main | 2023-09-26T12:56:47.765 DEBUG --- [ asgi_gw_0] l.s.l.i.docker_runtime_exe : Assigning container name of localstack-main-lambda-wrapped-async-916ddeabb4d4841522bc81436a4aa00c to executor 916ddeabb4d4841522bc81436a4aa00c
localstack_main | 2023-09-26T12:56:47.773 DEBUG --- [ asgi_gw_0] l.u.c.docker_sdk_client : Creating container with attributes: {'mount_volumes': [], 'ports':
Note : The invoke works when I set PROVIDER_OVERRIDE_LAMBDA=legacy in docker-compose, it takes ~18s but we do not want to use legacy.
Environment
- LocalStack: 2.2.1.dev
- Architecture: aarch64
Please let us know if you need any additional information. Thanks.
Hi @aryamohanan
Thank you for sharing the logs. The following warning message indicates that the Lambda function times out during the runtime startup:
localstack_main | 2023-09-26T12:56:57.767 WARN --- [ Thread-18] l.s.l.i.execution_environm : Execution environment 916ddeabb4d4841522bc81436a4aa00c for function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT.
Given that your Lambda function takes ~18 seconds to start indicates that the default startup timeout (10s) might be insufficient. Could you try increasing LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=30
?
(sidenote: In the long term, there might be some potential to optimize the packaging of the Lambda function to reduce the long cold starts.)
Hi @joe4dev Thanks for the replay, I've configured LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=30, but unfortunately, it hasn't resolved my issue. I'm still encountering the same error.
@aryamohanan Could you try some extra long timeout such as LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=300
(just for debugging)?
Is the Lambda package particularly big? Does the Lambda function do something time-consuming outside the handler function (i.e., during runtime startup)?
@joe4dev! The lambda function is quite straightforward, as it solely entails the printing of a log message. The problem is specific to my computer, and when I make the same alterations, everything functions correctly on a machine with the x86_64 architecture.
@aryamohanan
The problem is specific to my computer,
You mentioned aarch64
, what computer/operating system is it?
Does the Lambda invocation work with LAMBDA_IGNORE_ARCHITECTURE=1
(see configuration https://docs.localstack.cloud/references/configuration/#lambda)?
Further suggestions:
Consider updating to the latest Docker version, which fixes several bugs (especially for macOS) https://docs.docker.com/desktop/release-notes/
Consider disabling "Features in development" in the Docker settings.
I think I'm having the same issue, I've spent quite a while debugging through localstack logs and eventually found that the container that localstack starts is failing to start, see these logs
'docker' 'start' -i 'localstack-main-lambda-my-function-30e0a8296e46a94a95eef080c1107fec' ✔
time="2023-10-03T20:43:22Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory"
time="2023-10-03T20:43:22Z" level=panic msg="Post \"http://172.17.0.1:4566/_localstack_lambda/30e0a8296e46a94a95eef080c1107fec/status/30e0a8296e46a94a95eef080c1107fec/ready\": dial tcp 172.17.0.1:4566: connect: connection refused" func=go.amzn.com/lambda/rapid.handleStart file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:473"
panic: (*logrus.Entry) 0xc00011a0e0
goroutine 55 [running]:
github.com/sirupsen/logrus.Entry.log({0xc00011a000, 0xc019388030, {0x0, 0x0, 0x0}, 0x0, 0x0, {0x0, 0x0}, 0x0, ...}, ...)
/home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x2de
github.com/sirupsen/logrus.(*Entry).Log(0xc00011a070, 0x0, {0xc0004a7c20?, 0x0?, 0x98ca00?})
/home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xa8
github.com/sirupsen/logrus.(*Logger).Log(0xc00011a000, 0x0, {0xc0004a7c20, 0x1, 0x1})
/home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:193 +0x65
github.com/sirupsen/logrus.(*Logger).Panic(...)
/home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:234
github.com/sirupsen/logrus.Panic(...)
/home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/exported.go:129
go.amzn.com/lambda/rapid.handleStart({0xc992b8, 0xc00039a140}, 0xc000146000, 0x0?, 0xc00013a630)
/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:473 +0x51a
go.amzn.com/lambda/rapid.start({0xc992b8?, 0xc00039a140}, 0xc000146000)
/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:688 +0x545
go.amzn.com/lambda/rapid.Start(0xc0003b2000)
/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/sandbox.go:72 +0xdfe
go.amzn.com/lambda/rapidcore.(*SandboxBuilder).Create(0xc0003a00f0)
/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapidcore/sandbox.go:209 +0xf8
created by main.main
/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/main.go:194 +0xa4a
The localstack logs indicate
Execution environment 30e0a8296e46a94a95eef080c1107fec for function arn:aws:lambda:eu-west-1:000000000000:function:my-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT.
my lambda code is about as simple as could be
exports.main = async (event) => {
return {
statusCode: 200,
headers: {
'Content-Type': 'application/json'
},
body: {
message: 'Hello World!'
}
};
};
I'm on Linux (manjaro)
Hey @joe4dev ,
I attempted to use LAMBDA_IGNORE_ARCHITECTURE=1, but it didn't seem to work for me. I'm using Rancher Desktop for container management. Interestingly, when I set PROVIDER_OVERRIDE_LAMBDA=legacy, the invocation works perfectly fine. However, we don't wanted use the legacy mode. I'm attaching the docker information for your reference
OS : macOs 14.0
Client: Version: 24.0.2-rd Context: rancher-desktop Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.11.0 Path: xxxxxxxxx compose: Docker Compose (Docker Inc.) Version: v2.19.0 Path: xxxxxxxxx Server: Containers: 28 Running: 18 Paused: 0 Stopped: 10 Images: 42 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 1fbd70374134b891f97ce19c70b6e50c7b9f4e0d runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7 init version: Security Options: seccomp Profile: builtin Kernel Version: 6.1.30-0-virt Operating System: Alpine Linux v3.18 OSType: linux Architecture: aarch64 CPUs: 6 Total Memory: 11.66GiB Name: lima-rancher-desktop ID: xxxxxxxxxxx Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
I think I'm having the same issue, I've spent quite a while debugging through localstack logs and eventually found that the container that localstack starts is failing to start, see these logs
'docker' 'start' -i 'localstack-main-lambda-my-function-30e0a8296e46a94a95eef080c1107fec' ✔ time="2023-10-03T20:43:22Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory" time="2023-10-03T20:43:22Z" level=panic msg="Post \"http://172.17.0.1:4566/_localstack_lambda/30e0a8296e46a94a95eef080c1107fec/status/30e0a8296e46a94a95eef080c1107fec/ready\": dial tcp 172.17.0.1:4566: connect: connection refused" func=go.amzn.com/lambda/rapid.handleStart file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:473" panic: (*logrus.Entry) 0xc00011a0e0 goroutine 55 [running]: github.com/sirupsen/logrus.Entry.log({0xc00011a000, 0xc019388030, {0x0, 0x0, 0x0}, 0x0, 0x0, {0x0, 0x0}, 0x0, ...}, ...) /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x2de github.com/sirupsen/logrus.(*Entry).Log(0xc00011a070, 0x0, {0xc0004a7c20?, 0x0?, 0x98ca00?}) /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xa8 github.com/sirupsen/logrus.(*Logger).Log(0xc00011a000, 0x0, {0xc0004a7c20, 0x1, 0x1}) /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:193 +0x65 github.com/sirupsen/logrus.(*Logger).Panic(...) /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:234 github.com/sirupsen/logrus.Panic(...) /home/runner/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/exported.go:129 go.amzn.com/lambda/rapid.handleStart({0xc992b8, 0xc00039a140}, 0xc000146000, 0x0?, 0xc00013a630) /home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:473 +0x51a go.amzn.com/lambda/rapid.start({0xc992b8?, 0xc00039a140}, 0xc000146000) /home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/start.go:688 +0x545 go.amzn.com/lambda/rapid.Start(0xc0003b2000) /home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapid/sandbox.go:72 +0xdfe go.amzn.com/lambda/rapidcore.(*SandboxBuilder).Create(0xc0003a00f0) /home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/rapidcore/sandbox.go:209 +0xf8 created by main.main /home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/main.go:194 +0xa4a
The localstack logs indicate
Execution environment 30e0a8296e46a94a95eef080c1107fec for function arn:aws:lambda:eu-west-1:000000000000:function:my-function:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT.
my lambda code is about as simple as could be
exports.main = async (event) => { return { statusCode: 200, headers: { 'Content-Type': 'application/json' }, body: { message: 'Hello World!' } }; };
I'm on Linux (manjaro)
Hi @willm
The message dial tcp 172.17.0.1:4566: connect: connection refused
indicates that the Lambda container cannot connect to LocalStack. This is likely a Docker networking issue. Please ensure that Lambda containers are in the same network as the LocalStack container. Notice that LAMBDA_DOCKER_NETWORK
does currently not support host
mode.
How do you start LocalStack? Feel free to share your LocalStack configuration (e.g., docker-compose.yml
) for feedback.
Hey @joe4dev ,
I attempted to use LAMBDA_IGNORE_ARCHITECTURE=1, but it didn't seem to work for me. I'm using Rancher Desktop for container management. Interestingly, when I set PROVIDER_OVERRIDE_LAMBDA=legacy, the invocation works perfectly fine. However, we don't wanted use the legacy mode. I'm attaching the docker information for your reference
OS : macOs 14.0 Docker details
Hi @aryamohanan
The network field Network: bridge host ipvlan macvlan null overlay
contains host
but I guess the host
network is not used in LocalStack (currently not supported for Lambda)?
You mentioned that "The problem is specific to my computer". Does it work for others in your team with "Rancher Desktop", "x86_64", "macOs 14.0"? I am trying to understand what's the distinguishing factor here.
Is there a specific reason you need hostname: localstack
in the docker-compose?
I noticed in the initial LocalStack logs that the Lambda functions uses the network 'network': 'nodejs_default'
. Did you configure a specific network?
@joe4dev OK sorry maybe it's a different problem then, I'm actually trying to fix/get to the bottom of a different issue so I'm running localstack directly via make start
. I don't want to hijack this issue, there's probably something I need to do to allow the docker containers that lambda spins up to talk back to my host machine.
@joe4dev OK sorry maybe it's a different problem then, I'm actually trying to fix/get to the bottom of a different issue so I'm running localstack directly via
make start
. I don't want to hijack this issue, there's probably something I need to do to allow the docker containers that lambda spins up to talk back to my host machine.
@willm I see. Feel free to reach out in our community Slack. In the meantime, these documentation pages might be helpful:
Let's focus on resolving the issue from @markmashworth and @aryamohanan
✅ The infinite error loop has been resolved in major Lambda improvements https://github.com/localstack/localstack/pull/8970.
🚧 If I understand correctly, the remaining problem is specific to one machine setup (maybe related to aarch64, Rancher Desktop, 🤔 ) but generally works in LocalStack (tested on another x86_64 machine). Feel free to correct me @aryamohanan
Sidenote: Unless deploying large Lambda packages, startup times such as "~18s" or "20s" appear exceedingly slow.
@joe4dev The lambda function is a simple function with minimal code; the issue is with only the invoke command; all other commands operate well. Even after removing the hostname from the docker-compose file, the problem persists. We need to run this in macOS M1 with rancher-desktop.
Hi @aryamohanan
I just tried Ranger Desktop (Version 1.10.0 (1.10.0)) on my M1 Macbook Pro (macOS 13.6) using our official docker-compose.yml with DEBUG=1
:
version: "3.8"
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
environment:
- DEBUG=1
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
I was able to successfully invoke some simple Nodejs and Python Lambda functions from our integration test suite against LocalStack with Ranger Desktop. I used the default configuration for Ranger Desktop, notably:
Volumes: reverse-sshfs
Network: Disabled experimental socket-vmnet
Emulation: QEMU
One thing I noticed having Docker Desktop installed (but stopped) at the same time: I had to use ~/.rd/bin/docker-compose
directly to avoid permission issues (I guess a symlink without a Docker Desktop installation would fix the issue as well).
Does the container within Docker have root privileges?
I noticed that the last line in your initial Lambda container logs ended right before dropping root privileges: "Process running as root user". This might indicate that the container fails to drop root privileges because it does not have them and then just hangs.
You could try to disable that parity feature by setting LAMBDA_DOCKER_FLAGS=-e LOCALSTACK_USER=""
Would it be possible to share a minimal reproducible example?
Hi there,
I am having the same issues using OSX M1 and encountered the following issue.
2023-10-17 09:33:24 time="2023-10-17T01:33:24Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory"
I installed localstack via homebrew, but I have since uninstalled and reinstalled using docker, but I am unsure of where to add ROVIDER_OVERRIDE_LAMBDA=legacy
as mentioned by @Sotatek-HaiTrieu2
Hi there,
I am having the same issues using OSX M1 and encountered the following issue.
2023-10-17 09:33:24 time="2023-10-17T01:33:24Z" level=warning msg="Cannot list external agents" func=go.amzn.com/lambda/agents.ListExternalAgentPaths file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/agents/agent.go:71" error="open /opt/extensions: no such file or directory"
I installed localstack via homebrew, but I have since uninstalled and reinstalled using docker, but I am unsure of where to add
ROVIDER_OVERRIDE_LAMBDA=legacy
as mentioned by @Sotatek-HaiTrieu2
Hi @shen-qin
Your log messages are expected if you do not use a custom Lambda extension. Can you please provide some more context such as a reproducible scenario and more detailed logs using DEBUG=1
?
The provider override is a LocalStack configuration but not recommended. The legacy Lambda implementation will be removed with LocalStack 3.0. Hence, we prioritize solving any blocking issues.
@joe4dev Sorry for the delay in responding; I'm still having problems despite having tried all you said above. I've attached my docker-compose file for your reference. In addition to the local stack, I'm running a couple more containers.
version: '2' services: zookeeper: image: wurstmeister/zookeeper ports: - 2181:2181 kafka: image: wurstmeister/kafka:2.12-2.2.1 ports: - 9092:9092 - 29092:29092 depends_on: - "zookeeper" hostname: kafka0 environment: KAFKA_LISTENERS: EXTERNAL://:9092,PLAINTEXT://:29092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:29092,EXTERNAL://localhost:9092 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CREATE_TOPICS: "test:1:1" KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 volumes: - /var/run/docker.sock:/var/run/docker.sock - ./bin:/nodejs-test-bin command: ["/nodejs-test-bin/wait-for-it.sh", "-s", "-t", "120", "zookeeper:2181", "--", "start-kafka.sh"] schema-registry: image: confluentinc/cp-schema-registry:4.1.0 hostname: schema-registry depends_on: - "kafka" ports: - "8081:8081" environment: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181" SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "PLAINTEXT://kafka0:29092" SCHEMA_REGISTRY_HOST_NAME: schema-registry localstack: container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}" image: localstack/localstack:latest hostname: localstack ports: - "4566:4566" # LocalStack Gateway - "4510-4559:4510-4559" # external services port range environment: - DEBUG=${DEBUG-} - DOCKER_HOST=unix:///var/run/docker.sock volumes: - "/var/run/docker.sock:/var/run/docker.sock"
localstack_main | 2023-10-18T15:00:19.977 WARN --- [ Thread-12] l.s.l.i.execution_environm : Execution environment f0c4c8121bfccb7ef7d52d7e2c2b387b for function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async-v3:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT. localstack_main | 2023-10-18T15:00:20.324 INFO --- [ asgi_gw_0] localstack.request.aws : AWS lambda.Invoke => 500 (ServiceException)
Everything works OK when I run localstack alone in docker-compose, but it does not function with other containers, although it works well with the x86_64 architecture. If you require any other information, please let me know.
@joe4dev Sorry for the delay in responding; I'm still having problems despite having tried all you said above. I've attached my docker-compose file for your reference. In addition to the local stack, I'm running a couple more containers.
Thank you for the update @aryamohanan. We are sorry to hear that the configuration on ARM still causes issues.
Can you please try again with the latest LocalStack image and DEBUG=1
(or even LS_LOG=trace
) for more detailed logging? In particular, I'm interested in the Lambda container logs. (Alternatively, the Lambda containers are retained when setting LAMBDA_REMOVE_CONTAINERS=0
)
Sidenote: Adding the volume mount - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
helps with caching.
Everything works OK when I run localstack alone in docker-compose, but it does not function with other containers, although it works well with the x86_64 architecture.
@aryamohanan Can you please clarify whether this interpretation is correct:
timed out during startup
What is the architecture of the Lambda function (x86_64 | arm64)?
@joe4dev Your interpretations are correct. Running LocalStack with other containers through a combined docker-compose fails on ARM with the Lambda function invocation. Create command and other get commands are working fine. Here is my detailed log with DEBUG=1.
localstack_main | 2023-10-19T09:52:32.586 DEBUG --- [ asgi_gw_1] l.s.l.i.assignment : Starting new environment
localstack_main | 2023-10-19T09:52:32.586 DEBUG --- [ asgi_gw_1] l.s.l.i.docker_runtime_exe : Creating service endpoint for function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async-v3:$LATEST executor 8159049be77df1b5a9632b167e59f293
localstack_main | 2023-10-19T09:52:32.586 DEBUG --- [ asgi_gw_1] l.s.l.i.docker_runtime_exe : Finished creating service endpoint for function arn:aws:lambda:us-east-2:000000000000:function:wrapped-async-v3:$LATEST executor 8159049be77df1b5a9632b167e59f293
localstack_main | 2023-10-19T09:52:32.586 DEBUG --- [ asgi_gw_1] l.s.l.i.docker_runtime_exe : Assigning container name of localstack-main-lambda-wrapped-async-v3-8159049be77df1b5a9632b167e59f293 to executor 8159049be77df1b5a9632b167e59f293
localstack_main | 2023-10-19T09:52:32.594 DEBUG --- [ asgi_gw_1] l.u.c.docker_sdk_client : Creating container with attributes: {'self':
Lambda function architectures: [ 'x86_64' ]
There are no more logs after Process running as root user.
@aryamohanan Could you please try again without dropping the root privileges in the container?
v0.1.19-pre
=> v0.1.22-pre
).LAMBDA_INIT_USER=root
to disable dropping root privilegesWe cannot reproduce the issue on our side.
@joe4dev Hi there, sorry for the delay, the following is my log message. Thank you for the help!
Hi @shen-qin 👋
@shen-qin Can you quickly confirm whether you are working on the same issue/same team as @aryamohanan ?
@shen-qin The most recent logs indicate that there is some other issue with the Lambda image:
fork/exec /var/task/bootstrap: no such file or directory
❓ Are you using some customized Lambda image that does not contain a bootstrap file?
❓ Can you share a minimal example to reproduce the issue? Feel free to link a Github repository with a minimal reproducer sample.
Hi @joe4dev,
Thank you for the prompt response, I am not on the same team as aryamohanan. I am using whatever lambda image that is out of the box, I did not make any modifications. Sure let me make 1 to demostrate.
Hi @joe4dev,
Thank you for the prompt response, I am not on the same team as aryamohanan. I am using whatever lambda image that is out of the box, I did not make any modifications. Sure let me make 1 to demostrate.
@shen-qin That would be very helpful. Thank you for the clarifications.
Encountering the same issue of fork/exec /var/task/bootstrap: no such file or directory
after setting PROVIDER_OVERRIDE_LAMBDA=legacy
. Otherwise, setting it to v2 leads to timeout
Using a zipped bootstrap file implemented in Rust. On localstack version 1.4.0
**Resolved on version 2.3.2
. Thanks @joe4dev
Encountering the same issue of
fork/exec /var/task/bootstrap: no such file or directory
after settingPROVIDER_OVERRIDE_LAMBDA=legacy
. Otherwise, setting it to v2 leads to timeoutUsing a zipped bootstrap file implemented in Rust. On localstack version
1.4.0
Hi @wyilio
Can you please try at least LocalStack version 2.3.2
or even better our latest RC for the upcoming 3.0 release this week?
You can start LocalStack with DEBUG=1
to get more detailed log output. Feel free to share your logs or ideally a minimal example that reproduces the timeout.
FYI: The legacy
Lambda provider is removed in LocalStack 3.0.
I'm having similar issues to those @aryamohanan is facing. I followed all the tips that @joe4dev gave. But without success so far.
version: "3.8"
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
image: localstack/localstack
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
environment:
- DEBUG=1
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
$ sudo chmod 666 /var/run/docker.sock
$ docker compose -f docker-compose.yml up localstack --build
$ aws --endpoint-url=http://localhost:4566 lambda create-function \
--function-name test-go \
--runtime go1.x \
--zip-file fileb://lambda.zip \
--handler lambda.main \
--role arn:aws:iam::000000000000:role/test-go
$ aws --endpoint-url=http://localhost:4566 lambda invoke \
--function-name test-go \
--cli-binary-format raw-in-base64-out \
--payload '{"body": "{\"num1\": \"10\", \"num2\": \"10\"}" }' output.txt
package main
import (
"errors"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
)
func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
return events.APIGatewayProxyResponse{
Body: fmt.Sprintf("Hello, %v", "World"),
StatusCode: 200,
}, nil
}
func main() {
lambda.Start(handler)
}
$ uname -a
Linux user 6.1.XX-X-MANJARO #1 ... x86_64 GNU/Linux
$ localstack --version
2.3.0
$ aws --version
aws-cli/2.13.33 Python/3.11.5 Linux/6.1.62-1-MANJARO source/x86_64.manjaro.23 prompt/off
{
"FunctionName": "test-go",
"FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:test-go",
"Runtime": "go1.x",
"Role": "arn:aws:iam::000000000000:role/test-go",
"Handler": "lambda.main",
"CodeSize": 164,
"Description": "",
"Timeout": 3,
"MemorySize": 128,
"LastModified": "2023-11-21T22:05:15.438170+0000",
"CodeSha256": "Usj+xGrrL7m2IWJseME1/l/7ivUl2WRaSxc0t7Q4MhQ=",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "0c2e932b-b3b2-4013-bedf-efdb9be91c88",
"State": "Pending",
"StateReason": "The function is being created.",
"StateReasonCode": "Creating",
"PackageType": "Zip",
"Architectures": [
"x86_64"
],
"EphemeralStorage": {
"Size": 512
},
"SnapStart": {
"ApplyOn": "None",
"OptimizationStatus": "Off"
},
"RuntimeVersionConfig": {
"RuntimeVersionArn": "arn:aws:lambda:us-east-1::runtime:8eeff65f6809a3ce81507fe733fe09b835899b99481ba22fd75b5a7338290ec1"
}
}
```
Attaching to localstack_main
localstack_main | LocalStack supervisor: starting
localstack_main | LocalStack supervisor: localstack process (PID 15) starting
localstack_main |
localstack_main | LocalStack version: 3.0.1.dev
localstack_main | LocalStack build date: 2023-11-21
localstack_main | LocalStack build git hash: f354fd43
localstack_main |
localstack_main | 2023-11-21T22:00:28.711 DEBUG --- [ MainThread] stevedore._cache : reading /root/.cache/python-entrypoints/34d3c11768fc3a8d98b3943afb64d7b0b3abb3186ed714ff524e4736c2519853
localstack_main | 2023-11-21T22:00:28.716 DEBUG --- [ MainThread] stevedore._cache : writing to /root/.cache/python-entrypoints/34d3c11768fc3a8d98b3943afb64d7b0b3abb3186ed714ff524e4736c2519853
localstack_main | 2023-11-21T22:00:28.717 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='_patch_botocore_json_parser', value='localstack.aws.client:_patch_botocore_json_parser', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.718 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='_publish_config_as_analytics_event', value='localstack.runtime.analytics:_publish_config_as_analytics_event', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.718 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='_publish_container_info', value='localstack.runtime.analytics:_publish_container_info', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.718 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='_run_init_scripts_on_start', value='localstack.runtime.init:_run_init_scripts_on_start', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.719 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='deprecation_warnings', value='localstack.plugins:deprecation_warnings', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.719 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='register_partition_adjusting_proxy_listener', value='localstack.plugins:register_partition_adjusting_proxy_listener', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.719 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='setup_dns_configuration_on_host', value='localstack.dns.plugins:setup_dns_configuration_on_host', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.720 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='start_dns_server', value='localstack.dns.plugins:start_dns_server', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.720 DEBUG --- [ MainThread] stevedore.extension : found extension EntryPoint(name='validate_configuration', value='localstack.services.lambda_.plugins:validate_configuration', group='localstack.hooks.on_infra_start')
localstack_main | 2023-11-21T22:00:28.721 DEBUG --- [ MainThread] plugin.manager : instantiating plugin PluginSpec(localstack.hooks.on_infra_start._patch_botocore_json_parser =
An error occurred (ServiceException) when calling the Invoke operation (reached max retries: 2): Internal error while executing lambda
I'm having similar issues to those @aryamohanan is facing. I followed all the tips that @joe4dev gave. But without success so far.
Hi @sineto 👋
Thank you for sharing your details. What is currently missing is the build step.
I created a working sample with a few adjustments: https://github.com/joe4dev/localstack-lambda-golang1-sample
--handler lambda.main
=> --handler main
according to the buildThe AWS documentation about Building Lambda functions with Go provides more details on how to build and package Golang Lambda functions.
Note that the go1.x
runtime is deprecated and will reach EOL by the end of this year. Check out the AWS migration guide: https://aws.amazon.com/blogs/compute/migrating-aws-lambda-functions-from-the-go1-x-runtime-to-the-custom-runtime-on-amazon-linux-2/
@sineto Does the sample work for you? Feel free to adjust the sample or adapt it to your scenario and share it if you can reproduce the issue.
A few unrelated setup comments:
localstack --version
== 2.3.0
shows that you are running an older CLI version. Please update your CLI: https://docs.localstack.cloud/getting-started/installation/localstack_main
with localstack-main
as the container name in container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
See LocalStack v3 release notes for more details.
Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.
Hi @joe4dev, I'm trying to run Rust-based lambda function locally with LocalStack, but getting this issue as well. I'm not sure if it's the build issue or LocalStack issue. I'm on macOS/M1, building a binary for provided.al2023
runtime via Docker. I pushed the reproduction including detailed log: https://github.com/alex35mil/localstack-rust-lambda-issue
I found the culprits. It was a combo of naming and environment issues when deploying but not building with cargo lambda
.
I am having a similar problem with Localstack 3.0.2 and Serverless 3.38, also on a M1Pro Mac. My lambda is using node 18 as its environment. I have the latest version of docker, arm64 version.
localstack-main | 2023-12-26T13:02:29.065 DEBUG --- [ Thread-22] l.u.c.docker_sdk_client : Stopping container: localstack-main-lambda-lambda-name-goes-here-d17013f096ba8f6ff727c8a8908ec335
localstack-main | 2023-12-26T13:02:29.680 DEBUG --- [ Thread-22] l.u.c.docker_sdk_client : Removing container: localstack-main-lambda-lambda-name-goes-here-d17013f096ba8f6ff727c8a8908ec335
localstack-main | 2023-12-26T13:02:29.795 DEBUG --- [ve:$LATEST_0] l.s.l.i.event_manager : Service exception in lambda arn:aws:lambda:us-west-2:000000000000:function:lambda-name-goes-here:$LATEST: Execution environment timed out during startup.
localstack-main | 2023-12-26T13:02:29.814 DEBUG --- [ asgi_gw_4] l.services.sqs.models : deleting message 5dfeab90-03e4-499d-a757-d98545c4c66f from queue arn:aws:sqs:us-west-2:949334387222:lambda-name-goes-here-c872282e7a48b2d0ef6c0febfc1f5b5a
localstack-main | 2023-12-26T13:02:31.609 DEBUG --- [functhread41] l.services.sqs.models : enqueueing delayed messages 27948f53-941c-4aac-be54-d7ebbe8f3cf0 into queue arn:aws:sqs:us-west-2:949334387222:lambda-name-goes-here-c872282e7a48b2d0ef6c0febfc1f5b5a
localstack-main | 2023-12-26T13:02:31.609 DEBUG --- [ asgi_gw_2] l.services.sqs.models : de-queued message SqsMessage(id=27948f53-941c-4aac-be54-d7ebbe8f3cf0,group=None) from arn:aws:sqs:us-west-2:949334387222:lambda-name-goes-here-c872282e7a48b2d0ef6c0febfc1f5b5a
My docker-compose.yml
looks like this:
version: '3.8'
services:
localstack:
image: localstack/localstack:latest
container_name: "${LOCALSTACK_DOCKER_NAME-localstack-main}"
dns: 127.0.0.1
ports:
- "443:443"
- '4566:4566'
environment:
- DEBUG=1
- LAMBDA_DOCKER_FLAGS=-e NODE_OPTIONS=--inspect-brk=0.0.0.0:9229 -p 9229:9229
- LAMBDA_INIT_USER=root
- LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=60
- DOCKER_HOST=unix:///var/run/docker.sock
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
I have tried with LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT
up to 300 with the same results
I also sometimes see things like this:
localstack-main | 2023-12-26T13:02:28.920 WARN --- [ Thread-22] l.s.l.i.execution_environm : Execution environment d17013f096ba8f6ff727c8a8908ec335 for function arn:aws:lambda:us-west-2:000000000000:function:lambda-name-goes-here:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT.
localstack-main | 2023-12-26T13:02:29.064 DEBUG --- [ Thread-22] l.s.l.i.execution_environm : Logs from the execution environment d17013f096ba8f6ff727c8a8908ec335 after startup timeout:
localstack-main | [lambda d17013f096ba8f6ff727c8a8908ec335] time="2023-12-26T13:01:29Z" level=debug msg="No code archives set. Skipping download." func=main.DownloadCodeArchives file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/codearchive.go:21"
localstack-main | [lambda d17013f096ba8f6ff727c8a8908ec335] time="2023-12-26T13:01:29Z" level=debug msg="DNS server disabled." func=main.RunDNSRewriter file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/awsutil.go:145"
I don't understand how to access the logs that are alluded to here Logs from the execution environment d17013f096ba8f6ff727c8a8908ec335
so I don't know how to 'Check for errors during the startup of your Lambda function', but I don't think it is even starting the lambda because the debugger never enters it.
I found the culprits. It was a combo of naming and environment issues when deploying but not building with
cargo lambda
.
We are happy to hear the Rust Lambda deployment issue is resolved @alex35mil ✨ Thank you for sharing @alex35mil 🙏🙏
I am having a similar problem with Localstack 3.0.2 and Serverless 3.38, also on a M1Pro Mac. My lambda is using node 18 as its environment. I have the latest version of docker, arm64 version.
localstack-main | 2023-12-26T13:02:29.065 DEBUG --- [ Thread-22] l.u.c.docker_sdk_client : Stopping container: localstack-main-lambda-lambda-name-goes-here-d17013f096ba8f6ff727c8a8908ec335 localstack-main | 2023-12-26T13:02:29.680 DEBUG --- [ Thread-22] l.u.c.docker_sdk_client : Removing container: localstack-main-lambda-lambda-name-goes-here-d17013f096ba8f6ff727c8a8908ec335 localstack-main | 2023-12-26T13:02:29.795 DEBUG --- [ve:$LATEST_0] l.s.l.i.event_manager : Service exception in lambda arn:aws:lambda:us-west-2:000000000000:function:lambda-name-goes-here:$LATEST: Execution environment timed out during startup. localstack-main | 2023-12-26T13:02:29.814 DEBUG --- [ asgi_gw_4] l.services.sqs.models : deleting message 5dfeab90-03e4-499d-a757-d98545c4c66f from queue arn:aws:sqs:us-west-2:949334387222:lambda-name-goes-here-c872282e7a48b2d0ef6c0febfc1f5b5a localstack-main | 2023-12-26T13:02:31.609 DEBUG --- [functhread41] l.services.sqs.models : enqueueing delayed messages 27948f53-941c-4aac-be54-d7ebbe8f3cf0 into queue arn:aws:sqs:us-west-2:949334387222:lambda-name-goes-here-c872282e7a48b2d0ef6c0febfc1f5b5a localstack-main | 2023-12-26T13:02:31.609 DEBUG --- [ asgi_gw_2] l.services.sqs.models : de-queued message SqsMessage(id=27948f53-941c-4aac-be54-d7ebbe8f3cf0,group=None) from arn:aws:sqs:us-west-2:949334387222:lambda-name-goes-here-c872282e7a48b2d0ef6c0febfc1f5b5a
My
docker-compose.yml
looks like this:version: '3.8' services: localstack: image: localstack/localstack:latest container_name: "${LOCALSTACK_DOCKER_NAME-localstack-main}" dns: 127.0.0.1 ports: - "443:443" - '4566:4566' environment: - DEBUG=1 - LAMBDA_DOCKER_FLAGS=-e NODE_OPTIONS=--inspect-brk=0.0.0.0:9229 -p 9229:9229 - LAMBDA_INIT_USER=root - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=60 - DOCKER_HOST=unix:///var/run/docker.sock volumes: - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack" - "/var/run/docker.sock:/var/run/docker.sock"
I have tried with
LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT
up to 300 with the same resultsI also sometimes see things like this:
localstack-main | 2023-12-26T13:02:28.920 WARN --- [ Thread-22] l.s.l.i.execution_environm : Execution environment d17013f096ba8f6ff727c8a8908ec335 for function arn:aws:lambda:us-west-2:000000000000:function:lambda-name-goes-here:$LATEST timed out during startup. Check for errors during the startup of your Lambda function and consider increasing the startup timeout via LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT. localstack-main | 2023-12-26T13:02:29.064 DEBUG --- [ Thread-22] l.s.l.i.execution_environm : Logs from the execution environment d17013f096ba8f6ff727c8a8908ec335 after startup timeout: localstack-main | [lambda d17013f096ba8f6ff727c8a8908ec335] time="2023-12-26T13:01:29Z" level=debug msg="No code archives set. Skipping download." func=main.DownloadCodeArchives file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/codearchive.go:21" localstack-main | [lambda d17013f096ba8f6ff727c8a8908ec335] time="2023-12-26T13:01:29Z" level=debug msg="DNS server disabled." func=main.RunDNSRewriter file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/cmd/localstack/awsutil.go:145"
I don't understand how to access the logs that are alluded to here
Logs from the execution environment d17013f096ba8f6ff727c8a8908ec335
so I don't know how to 'Check for errors during the startup of your Lambda function', but I don't think it is even starting the lambda because the debugger never enters it.
Hi @Bandes Thank you for reaching out. Let me share some comments regarding the configuration and log inspection.
LAMBDA_INIT_USER=root
?LAMBDA_DOCKER_FLAGS
is currently limited to a single port mapping. Hence, please ensure that only one Lambda instance is active at any time. Does the code work without debugging?To check the Lambda container logs, please use Docker utilities (e.g., Docker Desktop or Docker logs) to inspect the logs of the Lambda docker container. The ID of the "execution environment" (e.g., d17013f096ba8f6ff727c8a8908ec335
) is part of the Docker container name:
Hint: Using LAMBDA_REMOVE_CONTAINERS=0
, Lambda containers are not removed, which might be helpful for debugging.
👉 We are looking forward to hearing your feedback @Bandes 👂
Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.
Hi @joe4dev
I successfully invoked the Lambda function after configuring the LAMBDA_INIT_USER=root
on my M1 machine. However, I couldn't locate documentation for this variable. Could you provide more information about it?
I've included the configuration in the docker-compose file for your reference.
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
image: localstack/localstack:latest
ports:
- "4566:4566" # LocalStack Gateway
- "4510-4559:4510-4559" # external services port range
environment:
- DEBUG=${DEBUG-}
- DOCKER_HOST=unix:///var/run/docker.sock
- LAMBDA_INIT_USER=root
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
Thanks in advance.
Hi @joe4dev I successfully invoked the Lambda function after configuring the
LAMBDA_INIT_USER=root
on my M1 machine. However, I couldn't locate documentation for this variable. Could you provide more information about it?I've included the configuration in the docker-compose file for your reference.
localstack: container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}" image: localstack/localstack:latest ports: - "4566:4566" # LocalStack Gateway - "4510-4559:4510-4559" # external services port range environment: - DEBUG=${DEBUG-} - DOCKER_HOST=unix:///var/run/docker.sock - LAMBDA_INIT_USER=root volumes: - "/var/run/docker.sock:/var/run/docker.sock"
Thanks in advance.
Thank you for sharing your feedback @aryamohanan 🙏 That's a very interesting behavior 🤔
LAMBDA_INIT_USER
is (currently) an internal configuration for LocalStack developers and therefore not (yet) documented in our LocalStack configuration page. It is implemented here in our custom Golang lambda-runtime-init and was primarily used for internal testing. When setting it to root
, we do not switch from the default container user root
to the user sbx_user1051
(matching the environment at AWS). Hence, we do not drop root privileges.
tests.aws.services.lambda_.test_lambda.TestLambdaBehavior.test_runtime_introspection_x86
) indicate a parity difference: at real AWS, the user slicer
exists and owns /var/task
. Since we are not (yet) aware of any related issues, we skip creating this user for now to avoid introducing unnecessary delay.Hello @joe4dev , I appreciate the clarification.
I'm encountering an issue solely with the invocation, as the other actions are functioning properly. The lambda code runs successfully on an actual AWS Lambda, and even when I set PROVIDER_OVERRIDE_LAMBDA=legacy, the invocation performs without any issues. I have other containers running using the same docker-compose file. I am attaching reproducer,
zookeeper:
image: wurstmeister/zookeeper
platform: linux/amd64
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka:2.12-2.2.1
platform: linux/amd64
ports:
- 9092:9092
- 29092:29092
depends_on:
- "zookeeper"
hostname: kafka0
environment:
KAFKA_LISTENERS: EXTERNAL://:9092,PLAINTEXT://:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:29092,EXTERNAL://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_CREATE_TOPICS: "test:1:1,test-topic-1:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./bin:/nodejs-bin
command: ["/nodejs-bin/wait-for-it.sh", "-s", "-t", "120", "zookeeper:2181", "--", "start-kafka.sh"]
schema-registry:
image: confluentinc/cp-schema-registry:4.1.0
platform: linux/amd64
hostname: schema-registry
depends_on:
- "kafka"
ports:
- "8081:8081"
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: "zookeeper:2181"
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: "PLAINTEXT://kafka0:29092"
SCHEMA_REGISTRY_HOST_NAME: schema-registry
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
image: localstack/localstack:latest
ports:
- "4566:4566" # LocalStack Gateway
- "4510-4559:4510-4559" # external services port range
environment:
- DEBUG=${DEBUG-}
- DOCKER_HOST=unix:///var/run/docker.sock
- LAMBDA_INIT_USER=root
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
Lambda function creation code.
const { LambdaClient} = require('@aws-sdk/client-lambda');
const lambdaClient = new LambdaClient({
endpoint: xxxxxxxxx,
region: 'us-east'
});
const lambdaFunctionCode = `
exports.handler = async (event) => {
const response = {
statusCode: 200,
body: JSON.stringify({ message: 'Hello, Lambda!' }),
};
return response;
};
`;
exports.createFunction = async functionName => {
zip.addFile('index.js', Buffer.from(lambdaFunctionCode));
const zipBuffer = zip.toBuffer();
const createFunctionParams = {
FunctionName: functionName,
Runtime: 'nodejs18.x',
Role: 'test/lambda-role',
Handler: 'index.handler',
Code: {
ZipFile: zipBuffer
}
};
await lambdaClient.send(new CreateFunctionCommand(createFunctionParams));
}
Please let me know if you need any other information.
Hello @joe4dev , I appreciate the clarification.
I'm encountering an issue solely with the invocation, as the other actions are functioning properly. The lambda code runs successfully on an actual AWS Lambda, and even when I set PROVIDER_OVERRIDE_LAMBDA=legacy, the invocation performs without any issues. I have other containers running using the same docker-compose file. I am attaching reproducer,
...
Please let me know if you need any other information.
Thank you for sharing @aryamohanan
Unfortunately, the reproducer is not fully self-contained, despite a few modifications:
// Modification 1: provide a working endpoint
const lambdaClient = new LambdaClient({
endpoint: 'http://127.0.0.1:4566',
region: 'us-east-1'
});
// Modification 2: invoke the deployment helper at the bottom
this.createFunction("my-lambda-function")
// Undefined dependencies
/Users/joe/Projects/LocalStack/issues/8878-lambda-timeout-node/deploy.js:17
zip.addFile('index.js', Buffer.from(lambdaFunctionCode));
^
ReferenceError: zip is not defined
at exports.createFunction (/Users/joe/Projects/LocalStack/issues/8878-lambda-timeout-node/deploy.js:17:3)
at Object.<anonymous> (/Users/joe/Projects/LocalStack/issues/8878-lambda-timeout-node/deploy.js:31:6)
at Module._compile (node:internal/modules/cjs/loader:1254:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
at Module.load (node:internal/modules/cjs/loader:1117:32)
at Module._load (node:internal/modules/cjs/loader:958:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
at node:internal/main/run_main_module:23:47
Node.js v18.16.0
Using the provided Lambda code within handler.js
:
exports.handler = async (event) => {
const response = {
statusCode: 200,
body: JSON.stringify({ message: 'Hello, Lambda!' }),
};
return response;
};
The following Makefile
successfully deploys make deploy
and invokes make invoke
the sample Nodejs Lambda function using the aws CLI:
package:
rm -f lambda.zip && zip lambda.zip handler.js
deploy: package
aws --endpoint-url http://localhost:4566 lambda create-function \
--function-name my-lambda-function \
--runtime nodejs18.x \
--zip-file fileb://lambda.zip \
--handler handler.handler \
--role arn:aws:iam::000000000000:role/lambda-role
invoke:
aws --endpoint-url=http://localhost:4566 lambda invoke \
--function-name my-lambda-function \
output.txt
Using this reference docker-compose.yml to start LocalStack (latest as of 2024-02-06):
version: "3.8"
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
image: localstack/localstack
ports:
- "127.0.0.1:4566:4566" # LocalStack Gateway
- "127.0.0.1:4510-4559:4510-4559" # external services port range
environment:
# LocalStack configuration: https://docs.localstack.cloud/references/configuration/
- DEBUG=${DEBUG:-0}
volumes:
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
Hello 👋! It looks like this issue hasn’t been active in longer than two weeks. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.
Is there an existing issue for this?
Current Behavior
When running localstack I am able to successfully create a layer and lambda function. However, when I try to invoke the function it times out. After the invocation, I see this same error get logged repeatedly by the container.
Expected Behavior
The lambda function should be invoked.
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
Here is my
docker-compose.yml
:Steps to reproduce:
I see the error mentioned above begin to log after running the invoke command.
My
lamba.zip
contains a single filehandler.py
that looks like this:Environment
Anything else?
No response