aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.48k stars 1.16k forks source link

Bug: Unable to Debug Python with sam local start-api --debug-port - Python debugger libraries not being copied #6546

Open antoineaws opened 7 months ago

antoineaws commented 7 months ago

Description:

Team, The SAM cli - sam local start-api --debug-port 5858 doesn't allow developers to debug their code (pause at breakpoints), even if --debug-function HelloWorldFunction is specified.

If you inspect the container that gets launched, note 2 things:

  1. We have the args between python3.11 and bootstrap.py. THe below image is what the args are when using vscode - launch.json - direct invoke. It also seems that the files needed for debugging e.g. debugpy.

    Screenshot 2024-01-10 at 4 54 28 PM
  2. The libraries needed for python debugging are also not copied e.g. debugpy etc. Below is what is copied with aws-sam direct invoke which is the correct files needed to debug:

    Screenshot 2024-01-11 at 10 40 58 AM

On the other hand, in the same project, that same command works great with NodeJS.

The expected behavior is to have the SAM lambda container wait on port --debug-port to allow IDE / VSCode to attach to it for debugging. Constantly having to launch direct-invoke tanks the developer experience and sam local start-api is a perfect tool for auto-reload and debugging.

Same with CDK:

Note, the same behavior happens with a CDK project where the command is run against the CDK synth template :sam local start-api -d 5858 -t ./cdk.out/template.yaml

Steps to reproduce:

Observed result:

bash-3.2$ sam local start-api -d 5858 --debug
2024-01-11 10:32:06,754 | Config file location: /Users/xxxx/Documents/Development/sam/sam-python/samconfig.toml                                                
2024-01-11 10:32:06,760 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in config file at              
'/Users/xxxx/Documents/Development/sam/sam-python/samconfig.toml'...                                                                                           
2024-01-11 10:32:06,763 | Configuration values successfully loaded.                                                                                               
2024-01-11 10:32:06,765 | Configuration values are: {'stack_name': 'sam-python'}                                                                                  
2024-01-11 10:32:06,793 | Using SAM Template at /Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml                                  
2024-01-11 10:32:06,891 | Using config file: samconfig.toml, config environment: default                                                                          
2024-01-11 10:32:06,893 | Expand command line arguments to:                                                                                                       
2024-01-11 10:32:06,894 | --template_file=/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml --debug_port=(5858,) --host=127.0.0.1  
--port=3000 --static_dir=public --layer_cache_basedir=/Users/xxxx/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1          
2024-01-11 10:32:07,034 | local start-api command is called                                                                                                       
2024-01-11 10:32:07,058 | No Parameters detected in the template                                                                                                  
2024-01-11 10:32:07,115 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction      
2024-01-11 10:32:07,116 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction1 is HelloWorldFunction1    
2024-01-11 10:32:07,118 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction2 is HelloWorldFunction2    
2024-01-11 10:32:07,120 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction3 is HelloWorldFunction3    
2024-01-11 10:32:07,122 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction4 is HelloWorldFunction4    
2024-01-11 10:32:07,124 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the       
resource id                                                                                                                                                       
2024-01-11 10:32:07,130 | 0 stacks found in the template                                                                                                          
2024-01-11 10:32:07,132 | No Parameters detected in the template                                                                                                  
2024-01-11 10:32:07,189 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction      
2024-01-11 10:32:07,190 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction1 is HelloWorldFunction1    
2024-01-11 10:32:07,192 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction2 is HelloWorldFunction2    
2024-01-11 10:32:07,194 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction3 is HelloWorldFunction3    
2024-01-11 10:32:07,196 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction4 is HelloWorldFunction4    
2024-01-11 10:32:07,198 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the       
resource id                                                                                                                                                       
2024-01-11 10:32:07,202 | 6 resources found in the stack                                                                                                          
2024-01-11 10:32:07,205 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'                                               
2024-01-11 10:32:07,207 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                                                               
/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml                                                                                  
2024-01-11 10:32:07,209 | Found Serverless function with name='HelloWorldFunction1' and CodeUri='HelloWorldFunction1'                                             
2024-01-11 10:32:07,212 | --base-dir is not presented, adjusting uri HelloWorldFunction1 relative to                                                              
/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml                                                                                  
2024-01-11 10:32:07,215 | Found Serverless function with name='HelloWorldFunction2' and CodeUri='HelloWorldFunction2'                                             
2024-01-11 10:32:07,218 | --base-dir is not presented, adjusting uri HelloWorldFunction2 relative to                                                              
/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml                                                                                  
2024-01-11 10:32:07,220 | Found Serverless function with name='HelloWorldFunction3' and CodeUri='HelloWorldFunction3'                                             
2024-01-11 10:32:07,221 | --base-dir is not presented, adjusting uri HelloWorldFunction3 relative to                                                              
/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml                                                                                  
2024-01-11 10:32:07,225 | Found Serverless function with name='HelloWorldFunction4' and CodeUri='HelloWorldFunction4'                                             
2024-01-11 10:32:07,228 | --base-dir is not presented, adjusting uri HelloWorldFunction4 relative to                                                              
/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/template.yaml                                                                                  
2024-01-11 10:32:07,244 | Found '1' API Events in Serverless function with name 'HelloWorldFunction'                                                              
2024-01-11 10:32:07,250 | Found '1' API Events in Serverless function with name 'HelloWorldFunction1'                                                             
2024-01-11 10:32:07,252 | Found '1' API Events in Serverless function with name 'HelloWorldFunction2'                                                             
2024-01-11 10:32:07,255 | Found '1' API Events in Serverless function with name 'HelloWorldFunction3'                                                             
2024-01-11 10:32:07,261 | Found '1' API Events in Serverless function with name 'HelloWorldFunction4'                                                             
2024-01-11 10:32:07,265 | Detected Inline Swagger definition                                                                                                      
2024-01-11 10:32:07,268 | Parsing Swagger document using 2.0 specification                                                                                        
2024-01-11 10:32:07,270 | Found '0' authorizers in resource 'ServerlessRestApi'                                                                                   
2024-01-11 10:32:07,272 | Lambda function integration not found in Swagger document at path='/nodejs-function-1' method='get'                                     
2024-01-11 10:32:07,276 | Lambda function integration not found in Swagger document at path='/nodejs-function-2' method='get'                                     
2024-01-11 10:32:07,280 | Lambda function integration not found in Swagger document at path='/python-function-2' method='get'                                     
2024-01-11 10:32:07,282 | Lambda function integration not found in Swagger document at path='/python-function-1' method='get'                                     
2024-01-11 10:32:07,284 | Lambda function integration not found in Swagger document at path='/hello' method='get'                                                 
2024-01-11 10:32:07,287 | Found '0' APIs in resource 'ServerlessRestApi'                                                                                          
2024-01-11 10:32:07,290 | Authorizer not found or disabled, returning early                                                                                       
2024-01-11 10:32:07,294 | Removed duplicates from '0' Explicit APIs and '5' Implicit APIs to produce '5' APIs                                                     
2024-01-11 10:32:07,298 | 5 APIs found in the template                                                                                                            
2024-01-11 10:32:07,327 | Mounting HelloWorldFunction4 at http://127.0.0.1:3000/python-function-2 [GET]                                                           
2024-01-11 10:32:07,332 | Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]                                                                        
2024-01-11 10:32:07,335 | Mounting HelloWorldFunction3 at http://127.0.0.1:3000/python-function-1 [GET]                                                           
2024-01-11 10:32:07,337 | Mounting HelloWorldFunction1 at http://127.0.0.1:3000/nodejs-function-1 [GET]                                                           
2024-01-11 10:32:07,340 | Mounting HelloWorldFunction2 at http://127.0.0.1:3000/nodejs-function-2 [GET]                                                           
2024-01-11 10:32:07,343 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your     
functions, changes will be reflected instantly/automatically. If you used sam build before running local commands, you will need to re-run sam build for the      
changes to be picked up. You only need to restart SAM CLI if you update your AWS SAM template                                                                     
2024-01-11 10:32:07,352 | Localhost server is starting up. Multi-threading = False                                                                                
2024-01-11 10:32:07 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
2024-01-11 10:32:07 Press CTRL+C to quit
2024-01-11 10:32:10,422 | Constructed Event 1.0 to invoke Lambda. Event: {'version': '1.0', 'httpMethod': 'GET', 'body': None, 'resource': '/hello',              
'requestContext': {'resourceId': '123456', 'apiId': '1234567890', 'resourcePath': '/hello', 'httpMethod': 'GET', 'requestId':                                     
'50e46488-415d-4925-a99f-487df64acb5c', 'accountId': '123456789012', 'stage': 'Prod', 'identity': {'apiKey': None, 'userArn': None, 'cognitoAuthenticationType':  
None, 'caller': None, 'userAgent': 'Custom User Agent String', 'user': None, 'cognitoIdentityPoolId': None, 'cognitoAuthenticationProvider': None, 'sourceIp':    
'127.0.0.1', 'accountId': None}, 'extendedRequestId': None, 'path': '/hello', 'protocol': 'HTTP/1.1', 'domainName': 'localhost:3000', 'requestTimeEpoch':         
1704987127, 'requestTime': '11/Jan/2024:15:32:07 +0000'}, 'queryStringParameters': None, 'multiValueQueryStringParameters': None, 'headers': {'Host':             
'localhost:3000', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0', 'Accept':                                 
'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate,  
br', 'Connection': 'keep-alive', 'Upgrade-Insecure-Requests': '1', 'Sec-Fetch-Dest': 'document', 'Sec-Fetch-Mode': 'navigate', 'Sec-Fetch-Site': 'cross-site',    
'X-Forwarded-Proto': 'http', 'X-Forwarded-Port': '3000'}, 'multiValueHeaders': {'Host': ['localhost:3000'], 'User-Agent': ['Mozilla/5.0 (Macintosh; Intel Mac OS X
10.15; rv:121.0) Gecko/20100101 Firefox/121.0'], 'Accept': ['text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'],             
'Accept-Language': ['en-US,en;q=0.5'], 'Accept-Encoding': ['gzip, deflate, br'], 'Connection': ['keep-alive'], 'Upgrade-Insecure-Requests': ['1'],                
'Sec-Fetch-Dest': ['document'], 'Sec-Fetch-Mode': ['navigate'], 'Sec-Fetch-Site': ['cross-site'], 'X-Forwarded-Proto': ['http'], 'X-Forwarded-Port': ['3000']},   
'pathParameters': None, 'stageVariables': None, 'path': '/hello', 'isBase64Encoded': False}                                                                       
2024-01-11 10:32:10,433 | Found one Lambda function with name 'HelloWorldFunction'                                                                                
2024-01-11 10:32:10,435 | Invoking app.lambda_handler (python3.11)                                                                                                
2024-01-11 10:32:10,436 | No environment variables found for function 'HelloWorldFunction'                                                                        
2024-01-11 10:32:10,437 | Loading AWS credentials from session with profile 'None'                                                                                
2024-01-11 10:32:10,461 | Resolving code path. Cwd=/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build,                                            
CodeUri=/Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/HelloWorldFunction                                                                     
2024-01-11 10:32:10,462 | Resolved absolute path to code is /Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/HelloWorldFunction                 
2024-01-11 10:32:10,464 | Code /Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/HelloWorldFunction is not a zip/jar file                        
2024-01-11 10:32:11,536 | Local image is up-to-date                                                                                                               
2024-01-11 10:32:11,553 | Checking free port on 127.0.0.1:8483                                                                                                    
2024-01-11 10:32:11,564 | Using local image: public.ecr.aws/lambda/python:3.11-rapid-x86_64.                                                                      

2024-01-11 10:32:11,566 | Mounting /Users/xxxx/Documents/Development/sam/sam-python/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime
container                                                                                                                                                         
2024-01-11 10:32:11,991 | Setting up SIGTERM interrupt handler                                                                                                    
{'body': None, 'headers': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.5', 'Connection': 'keep-alive', 'Host': 'localhost:3000', 'Sec-Fetch-Dest': 'document', 'Sec-Fetch-Mode': 'navigate', 'Sec-Fetch-Site': 'cross-site', 'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0', 'X-Forwarded-Port': '3000', 'X-Forwarded-Proto': 'http'}, 'httpMethod': 'GET', 'isBase64Encoded': False, 'multiValueHeaders': {'Accept': ['text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'], 'Accept-Encoding': ['gzip, deflate, br'], 'Accept-Language': ['en-US,en;q=0.5'], 'Connection': ['keep-alive'], 'Host': ['localhost:3000'], 'Sec-Fetch-Dest': ['document'], 'Sec-Fetch-Mode': ['navigate'], 'Sec-Fetch-Site': ['cross-site'], 'Upgrade-Insecure-Requests': ['1'], 'User-Agent': ['Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0'], 'X-Forwarded-Port': ['3000'], 'X-Forwarded-Proto': ['http']}, 'multiValueQueryStringParameters': None, 'path': '/hello', 'pathParameters': None, 'queryStringParameters': None, 'requestContext': {'accountId': '123456789012', 'apiId': '1234567890', 'domainName': 'localhost:3000', 'extendedRequestId': None, 'httpMethod': 'GET', 'identity': {'accountId': None, 'apiKey': None, 'caller': None, 'cognitoAuthenticationProvider': None, 'cognitoAuthenticationType': None, 'cognitoIdentityPoolId': None, 'sourceIp': '127.0.0.1', 'user': None, 'userAgent': 'Custom User Agent String', 'userArn': None}, 'path': '/hello', 'protocol': 'HTTP/1.1', 'requestId': '50e46488-415d-4925-a99f-487df64acb5c', 'requestTime': '11/Jan/2024:15:32:07 +0000', 'requestTimeEpoch': 1704987127, 'resourceId': '123456', 'resourcePath': '/hello', 'stage': 'Prod'}, 'resource': '/hello', 'stageVariables': None, 'version': '1.0'}
END RequestId: c0ef6397-385f-453b-b8f1-038713569e75
REPORT RequestId: c0ef6397-385f-453b-b8f1-038713569e75  Init Duration: 0.08 ms  Duration: 124.43 ms     Billed Duration: 125 ms Memory Size: 128 MB     Max Memory Used: 128 MB

2024-01-11 10:32:12,343 | Cleaning all decompressed code dirs                                                                                                     
2024-01-11 10:32:12,346 | No Content-Type given. Defaulting to 'application/json'.                                                                                
2024-01-11 10:32:12 127.0.0.1 - - [11/Jan/2024 10:32:12] "GET /hello HTTP/1.1" 200 -

Expected result:

Expected that the sam lambda container will wait using debugpy for the IDE attach to it and then allow a developer to debug.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Mac OS Ventura 13.6.3
  2. sam --version: 1.106.0
  3. AWS region: ca-central-1
{
  "version": "1.106.0",
  "system": {
    "python": "3.8.13",
    "os": "macOS-13.6.3-x86_64-i386-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.7",
    "aws_cdk": "2.110.0 (build c6471f2)",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Feel free if you need more info. Thanks!

lucashuy commented 7 months ago

Hi, to attach a debugger manually to the container, you'll have to provide the debugging arguments yourself.

Using the hello world Python 3.8 template, I added debugpy to the existing requirements file. Then after sam build, I ran sam local start-api -d 9999 --debug --debug-args "-m debugpy --listen 0.0.0.0:9999" to start the API.

After waiting for the API to start, I ran curl localhost:3000/hello and the API waited for me to attach a debugger. Using the following VSCode configuration:

{
    "name": "Python: Remote Attach",
    "type": "python",
    "request": "attach",
    "connect": {
        "host": "localhost",
        "port": 9999
    },
    "pathMappings": [
        {
            "localRoot": "${workspaceFolder}/hello_world",
            "remoteRoot": "/var/task"
        }
    ],
    "justMyCode": false
},

I was able to set a break point and debug the function.

The AWS Toolkit extension abstracts away all this setup as you've shown. Let me know if you aren't able to get the debugger working, or if it still refuses to wait for you to attach a debugging session.

antoineaws commented 7 months ago

@lucashuy, I will try it out shortly and report back, but can we enhance the sam local start-api and make these steps (add debugpy and start it) trigger automatically? Similar to how NodeJS seems to handle it. Cheers

antoineaws commented 7 months ago

@lucashuy, I tried setting the lambda runtime in the SAM template to python3.8 and python 3.11. I then ran sam build.

For both runtimes, the debugpy module doesn't get triggered. The behavior is that the function executes without pausing for vscode to attach. Inspecting the provisioned container in docker, shows that the arguments didn't propagate

Screenshot 2024-01-12 at 3 46 50 PM

Can you share which version of the sam cli and docker engine you tested with? Thanks

lucashuy commented 7 months ago

Hi, sorry for missing this, but the --wait-for-client should be provided as well.

Using SAM CLI 1.107.0 and Docker Engine 24.0.6 I can get the debugger to wait:

sam local start-api -d 9999 --debug --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
{
    "Id": "44963bd2dff4803275ece386662aaa0cb80219f9e0d2d0458fa5861e07a5c7e7",
    "Created": "2024-01-12T23:39:14.668381755Z",
    "Path": "/var/rapid/aws-lambda-rie",
    "Args": [
        "--log-level",
        "error",
        "/var/lang/bin/python3.8",
        "-m",
        "debugpy",
        "--listen",
        "0.0.0.0:9999",
        "--wait-for-client",
        "/var/runtime/bootstrap.py"
    ],
        ...
        ...

This is after adding debugpy as a requirement for the Lambda function.

antoineaws commented 7 months ago

Hi @lucashuy , this worked with caveats noted below:

  1. Each function require an entry in launch.json - In launch.json, you must point pathMappings => localRoot to the app.py folder otherwise the breakpoint won't get hit. Which means that if you have 10 lambda functions you will need 10 definitions in launch.json, does that make sense? With NodeJS the behavior is different, I use 1 remote attach definition in launch.json and it works with all functions regardless of where "localRoot" is pointing to.
  2. CDK Projects require sam build - For CDK projects, you need to execute sam build -t cdk.out/template.yml to generate .aws-sam artifacts, otherwise you debugpy won't be copied even if you had a .venv
  3. Auto-reload requires sam build - With this approach, auto reload doesn't pick up the changes and developer would need to do a sam build.

Can we pass the --debug-args transparently when -d 5858 is set and python lambdas are detected?

bettlebrox commented 4 months ago

I could also get the debugger to wait using --debug-args but it wasn't intuitive given other language debuggers don't require this or adding a dependancy.

An added complexity was that I was including my lambda requirements through a aws_cdk.aws_lambda_python_alpha.PythonLayerVersion but adding debugpy here didn't work - I got a module not found error - I suppose /opt/python is added to the path after the debug args are executed. I had to pip install debugpy -t lambda where lambda is the root of the AssetCode.

@antoineaws I no longer experiencing any of the caveats you mentioned.

SAM CLI 1.113.0 CDK 2.133.0

CDK construct: getThemes = lambda_.Function( self, "getThemes", runtime=lambda_.Runtime.PYTHON_3_8, code=lambda_.AssetCode.from_asset(path.join(os.getcwd(), "python/lambda")), handler="get_themes.lambda_handler", vpc=vpc, layers=[reqs_layer], tracing=lambda_.Tracing.ACTIVE, timeout=Duration.seconds(45), )

SAM start-api command: sam local start-api -t ./cdk.out/NameStack-nwbxl.template.json --warm-containers eager --env-vars python/lambda/local.json -d 9999 --debug-function getNameFA47 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"

skurihara-cw commented 3 months ago

Hi, sorry for missing this, but the --wait-for-client should be provided as well.

Using SAM CLI 1.107.0 and Docker Engine 24.0.6 I can get the debugger to wait:

sam local start-api -d 9999 --debug --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
{
  "Id": "44963bd2dff4803275ece386662aaa0cb80219f9e0d2d0458fa5861e07a5c7e7",
  "Created": "2024-01-12T23:39:14.668381755Z",
  "Path": "/var/rapid/aws-lambda-rie",
  "Args": [
      "--log-level",
      "error",
      "/var/lang/bin/python3.8",
      "-m",
      "debugpy",
      "--listen",
      "0.0.0.0:9999",
      "--wait-for-client",
      "/var/runtime/bootstrap.py"
  ],
        ...
        ...

This is after adding debugpy as a requirement for the Lambda function.

I tried it, but it outputs an error.

Extract from sam local start-api log.

START RequestId: ac42975a-b8d7-42df-8c9a-4461e23e8fae Version: $LATEST
14 May 2024 08:22:13,104 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: killed InvokeID=
14 May 2024 08:22:13,116 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: killed InvokeID=f7e10428-774e-4580-b2f5-206259b0 f4a8
14 May 2024 08:22:13,134 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-14 17:22:13,082 | Failed to deserialize response from RIE, returning the raw response as is

2024-05-14 17:22:14,091 | Invalid lambda response received: Lambda response must be valid json
2024-05-14 17:22:14,093 | Lambda execution failed ()
2024-05-14 17:22:14 127.0.0.1 - - [14/May/2024 17:22:14] "GET /hello HTTP/1.1" 502 -

sam --version SAM CLI, version 1.116.0

@antoineaws Can you tell me the launch.json and sam local command for the success?


I am aware that the debugger mode is probably not ready to wait in the first place...

https://github.com/aws/aws-sam-cli/assets/123167609/028a43d4-2527-4f5c-ac4e-0fcba0a96f17

antoineaws commented 3 months ago

@skurihara-cw The sam command is similar to what @lucashuy suggested here: https://github.com/aws/aws-sam-cli/issues/6546#issuecomment-1890148136 Here's my launch.json: (note direct invoke and attach)

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "aws-sam",
            "request": "direct-invoke",
            "name": "Python: Function 1 + Payload",
            "invokeTarget": {
                "target": "code",
                "projectRoot": "${workspaceFolder}/xxx/functions/python-function-1",
                "lambdaHandler": "app.lambda_handler"
            },
            "lambda": {
                "runtime": "python3.9",
                "payload": {
                    "path": "${workspaceFolder}/xxx/functions/events/apigw-event.json"
                },
                "environmentVariables": {}
            },
            "aws": {
                "credentials": "profile:my_profile",
                "region": "ca-central-1"
            }
        },
        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "host": "127.0.0.1",
            "port": 5858,
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/xxxx/functions/python-function-2",
                    "remoteRoot": "/var/task"
                }
            ],
            "justMyCode": true
        }
    ]
}
jpangburn commented 3 months ago

@lucashuy Bravo for spelling out in detail how to make this work!!! So many "solutions" out there lacking critical details.

@bettlebrox Thank you as well for spelling out in detail how to make it work for CDK, as well as that it doesn't require a "sam build" every time you change code.

For any other rookies (like me) who followed the CDK serverless tutorial in the docs at https://docs.aws.amazon.com/cdk/v2/guide/serverless_example.html, this is everything I had to do beyond what the tutorial tells you to do in order to get remote debugging going for python. First, my final directory structure for context:

image

You also need to have Docker installed, and follow the instructions at https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html to install the AWS SAM CLI. The other docs in that SAM guide are a helpful reference.

The CDK serverless example tutorial gives you a JS version of the HelloWorld lambda function, but you want python. In the top of your project, create the hello_lambda directory you see in my project. Create an empty __init__.py under that (as shown), then create app.py (highlighted in the screenshot) with these contents (the silly return_message stuff is to have some lines to test debugging):

import json

def lambda_handler(event, context):
    return_message = "hello"
    return_message += " world!"

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": return_message,
        }),
    }

To finish switching from Node.JS to Python for the lambda function, edit the cdk_hello_world_stack.py file and replace the hello_world_function declaration that defines the Node.JS version of hello world to this:

    hello_world_function = aws_lambda.Function(
        self,
        "HelloWorldFunction",
        runtime=aws_lambda.Runtime.PYTHON_3_9,
        handler='app.lambda_handler',
        code=aws_lambda.Code.from_asset('./hello_lambda')
    )

Later for debugging, you'll need the debugpy module in this same directory (seems like there's a bug that SAM doesn't put it in the target Docker container for you). As @bettlebrox said, do it like this python3 -m pip install debugpy -t hello_lambda- as my screenshot showed, it should be right there in your hello_lambda directory.

Next (still at the top level in the command line), run cdk synth --no-staging. Now you need the unique name it assigned to the function (it might be called the logical id, but I'm a noob so I don't know). You can find this in the mess it spits out to the terminal in a section like this:

HelloWorldFunctionB2AB6E79: Type: AWS::Lambda::Function

In my case it wasn't far from the top. You can also find it in cdk.out/CdkHelloWorldStack.template.json, for me it was the second object under Resources and looked like this:

"HelloWorldFunctionB2AB6E79": { "Type": "AWS::Lambda::Function",

In either case, you take that name and put it into the command that @bettlebrox gave us, like this:

sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"

At this point, your server should be started and waiting for you to send it HTTP requests. Give it a try with a second terminal window like this curl http://127.0.0.1:3000/hello. It should sit there and not respond. It's waiting for you to connect a debugger. If it errors, or returns something right away- I'm sorry but something has gone wrong, fix it before you proceed.

Create a .vscode/launch.json file, and give it the following contents (or edit your existing one to include this configuration):

{ // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: Remote Attach hello lambda", "type": "debugpy", "request": "attach", "connect": { "host": "localhost", "port": 9999 }, "pathMappings": [ { "localRoot": "${workspaceFolder}/hello_lambda", "remoteRoot": "/var/task" } ], "justMyCode": false } ] }

Set a breakpoint in the app.py on one of those return_message lines. In your "Run and Debug" tab in VSCode, run this debugger. If all is correct, your debugger should break immediately on your breakpoint. Continuing should let the function finish and you should see the response in your second terminal window. You can now edit the app.py file, save it, and issue a new HTTP request without doing anything else at the command line. If you edit the file, you will have to relaunch the debugger (it's because the docker container restarts with the new file contents so disconnects the debugger). If you don't edit the file and just want to debug again, it'll just work.

skurihara-cw commented 3 months ago

@jpangburn

sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"

Are the following options always necessary? Can they be supplemented?🥺

--warm-containers eager --debug-function HelloWorldFunctionB2AB6E79

jpangburn commented 3 months ago

@jpangburn

sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"

Are the following options always necessary? Can they be supplemented?🥺

--warm-containers eager --debug-function HelloWorldFunctionB2AB6E79

@skurihara-cw I wondered the same thing myself and tested them but felt like my previous comment was too long already, so left it out.

For eager/lazy, I noticed after you run/edit the function it always says Lambda Function 'HelloWorldFunction' source code has been changed, terminate its warm container. The new container will be created in lazy mode. So after the first run it definitely doesn't matter because then it's lazy no matter what you said. On the first run though, in eager mode if you try to connect your debugger before you make the API call you don't get an error message- just the debugger doesn't connect, but in lazy mode you get an error message you have to close (connection refused). In all cases, you can't connect the debugger after the API call until a few seconds have passed and the container accepts the connection. So I just left it in case @bettlebrox knew something I didn't. I suspect in other versions of SAM it might be necessary because the documentation says it's tied to the --debug-function parameter. Again, I'm a noob here, I started with CDK yesterday, and tried SAM once some years ago so take everything I say with that in mind.

The --debug-function parameter is, unfortunately, critical. When I tested it, without that function the API call completes without waiting and you cannot connect the debugger. It appears tied to the --debug-port aka -d and --debug-args options. I think what's happening is that each lambda function in your CDK is deployed to a docker container and only one container can be debugging at a time (obviously because you can only open the same port once), so I think you need that --debug-function to tell SAM which container to open the debug port on? That's a guess though.

Why you need the -d 9999 and also having --listen 0.0.0.0:9999 in the --debug-args, I don't know and didn't try to remove it. This thing was so hard to get debugging, mostly I was just relieved it finally worked 😅 so I only tried the above two things you happened to ask about. I tried it now out of curiosity, and removing either of those switches with 9999 did not work. I suspect one is for SAM to listen locally (the former one) and the latter one is for debugpy inside the container to listen on (SAM must connect to that container somehow and make the local port available to connect to from VSCode). In any case, they are required.

bettlebrox commented 3 months ago

@jpangburn

sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"

Are the following options always necessary? Can they be supplemented?🥺 --warm-containers eager --debug-function HelloWorldFunctionB2AB6E79

@skurihara-cw I wondered the same thing myself and tested them but felt like my previous comment was too long already, so left it out.

For eager/lazy, I noticed after you run/edit the function it always says Lambda Function 'HelloWorldFunction' source code has been changed, terminate its warm container. The new container will be created in lazy mode. So after the first run it definitely doesn't matter because then it's lazy no matter what you said. On the first run though, in eager mode if you try to connect your debugger before you make the API call you don't get an error message- just the debugger doesn't connect, but in lazy mode you get an error message you have to close (connection refused). In all cases, you can't connect the debugger after the API call until a few seconds have passed and the container accepts the connection. So I just left it in case @bettlebrox knew something I didn't. I suspect in other versions of SAM it might be necessary because the documentation says it's tied to the --debug-function parameter. Again, I'm a noob here, I started with CDK yesterday, and tried SAM once some years ago so take everything I say with that in mind.

The --debug-function parameter is, unfortunately, critical. When I tested it, without that function the API call completes without waiting and you cannot connect the debugger. It appears tied to the --debug-port aka -d and --debug-args options. I think what's happening is that each lambda function in your CDK is deployed to a docker container and only one container can be debugging at a time (obviously because you can only open the same port once), so I think you need that --debug-function to tell SAM which container to open the debug port on? That's a guess though.

Why you need the -d 9999 and also having --listen 0.0.0.0:9999 in the --debug-args, I don't know and didn't try to remove it. This thing was so hard to get debugging, mostly I was just relieved it finally worked 😅 so I only tried the above two things you happened to ask about. I tried it now out of curiosity, and removing either of those switches with 9999 did not work. I suspect one is for SAM to listen locally (the former one) and the latter one is for debugpy inside the container to listen on (SAM must connect to that container somehow and make the local port available to connect to from VSCode). In any case, they are required.

I had a similar experience tinkering until I got something to work so I just copied in what was working for me. I believe --debug-function is required for debugging but --warm-containers is not. Thats just an assumption I haven't tested it. Also I used 9999 port for debugging because I was debugging something else on the default port so I think both those options can be removed. Again not tested.