aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.52k stars 1.17k forks source link

Initializing the lambda functions containers is so slow #3025

Closed aleCeres closed 2 years ago

aleCeres commented 3 years ago

I was trying to run aws sam in my local environment using the following command:

sam local start-api --docker-network my_docker_network --port 4000 --warm-containers EAGER

this is what I see in the log

Initializing the lambda functions containers.
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
Building image...Building image...Building image...Building image...Building image...RuntimeDependenciesLayer is a local Layer in the template
Building image...Building image...Building image...RuntimeDependenciesLayer is a local Layer in the template
RuntimeDependenciesLayer is a local Layer in the template
Building image...Building image...Building image...Building image............................................................................................

So I have implemented 10 api (10 lambdas) but the initialization time to build the containers is about 7 minutes.

Noderuntime: node14.x SAM CLI, version 1.23.0

Any advise to improve build time of lambda function containers? Thanks in advance.

CoshUS commented 3 years ago

Can you provide log with --debug flag specified and also your docker version? I tried to reproduce this with similar number of Image type functions and they finished within a minute. One possible workaround is to use --warm-containers LAZY and defer emulation image building to later when functions are invoked.

aleCeres commented 3 years ago

Sure:

2021-07-09 16:36:00,972 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container .. 2021-07-09 16:36:01,474 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:01,475 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container .......... 2021-07-09 16:36:07,602 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:07,602 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container .. .. 2021-07-09 16:36:08,261 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:08,261 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container 2021-07-09 16:36:08,328 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:08,328 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container .. 2021-07-09 16:36:08,947 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:08,947 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container ... . 2021-07-09 16:36:11,872 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:11,872 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container 2021-07-09 16:36:11,891 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:11,892 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container .. 2021-07-09 16:36:12,422 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:12,422 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container .. ..2021-07-09 16:36:12 Connection pool is full, discarding connection: localhost

2021-07-09 16:36:12,548 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:12,549 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container ..2021-07-09 16:36:12 Connection pool is full, discarding connection: localhost

..2021-07-09 16:36:12 Connection pool is full, discarding connection: localhost

2021-07-09 16:36:12,714 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:12,716 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container 2021-07-09 16:36:12,717 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:12,718 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container 2021-07-09 16:36:12,762 | Skip pulling image and use local one: samcli/lambda:nodejs14.x-f88a901bbe51b578e9260ae49.

2021-07-09 16:36:12,762 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container 2021-07-09 16:36:12,762 | Mounting path_to_repo as /var/task:ro,delegated inside runtime container 2021-07-09 16:38:06,353 | Async execution completed 2021-07-09 16:38:06,354 | Containers Initialization is done. 2021-07-09 16:38:06,354 | No Parameters detected in the template 2021-07-09 16:38:06,395 | No Parameters detected in the template 2021-07-09 16:38:06,432 | Found '1' API Events in Serverless function with name 'f1' 2021-07-09 16:38:06,432 | Found '1' API Events in Serverless function with name 'f2' 2021-07-09 16:38:06,432 | Found '1' API Events in Serverless function with name 'f3' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f4' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f5' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f6' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f7' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f8' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f9' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f10' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f11' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f12' 2021-07-09 16:38:06,433 | Found '1' API Events in Serverless function with name 'f13' 2021-07-09 16:38:06,433 | Detected Inline Swagger definition 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f1' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f2' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f3' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f4' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f5' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f6' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f7' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f8' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f9' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f10' method='post' 2021-07-09 16:38:06,433 | Lambda function integration not found in Swagger document at path='/f11' method='post' 2021-07-09 16:38:06,434 | Lambda function integration not found in Swagger document at path='/f12' method='post' 2021-07-09 16:38:06,434 | Lambda function integration not found in Swagger document at path='/f13' method='post' 2021-07-09 16:38:06,434 | Found '0' APIs in resource 'ServerlessRestApi' 2021-07-09 16:38:06,434 | Removed duplicates from '0' Explicit APIs and '13' Implicit APIs to produce '13' APIs 2021-07-09 16:38:06,434 | 13 APIs found in the template 2021-07-09 16:38:06,444 | Mounting f7 at http://127.0.0.1:4000/f1 [POST] 2021-07-09 16:38:06,444 | Mounting f8 at http://127.0.0.1:4000/f2 [POST] 2021-07-09 16:38:06,444 | Mounting f13 at http://127.0.0.1:4000/f3 [POST] 2021-07-09 16:38:06,444 | Mounting f4 at http://127.0.0.1:4000/f4 [POST] 2021-07-09 16:38:06,444 | Mounting f5 at http://127.0.0.1:4000/f5 [POST] 2021-07-09 16:38:06,444 | Mounting f10 at http://127.0.0.1:4000/f6 [POST] 2021-07-09 16:38:06,444 | Mounting f2 at http://127.0.0.1:4000/f7 [POST] 2021-07-09 16:38:06,444 | Mounting f1 at http://127.0.0.1:4000/f8 [POST] 2021-07-09 16:38:06,444 | Mounting f3 at http://127.0.0.1:4000/f9 [POST] 2021-07-09 16:38:06,444 | Mounting f12 at http://127.0.0.1:4000/f10 [POST] 2021-07-09 16:38:06,474 | Mounting f9 at http://127.0.0.1:4000/f11 [POST] 2021-07-09 16:38:06,474 | Mounting f11 at http://127.0.0.1:4000/f12 [POST] 2021-07-09 16:38:06,474 | Mounting f6 at http://127.0.0.1:4000/f13 [POST] 2021-07-09 16:38:06,474 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template 2021-07-09 16:38:06,474 | Localhost server is starting up. Multi-threading = True 2021-07-09 16:38:06 * Running on http://127.0.0.1:4000/ (Press CTRL+C to quit)



And If I try to use `--warn-containers LAZY` the image building time when single function is invoked is still high.
CoshUS commented 3 years ago

I tried to reproduce it locally again with 15 nodejs14 functions but still didn't encounter the long image build time. It seems the warning Connection pool is full, discarding connection: localhost might be related. Can you try some of the solution in this post?

If the issue still persists, can you tell me more about your

jfuss commented 2 years ago

Closing given no response. If you are still encountering this, please create a new issue (closed issues are hard to track)