Open antoineaws opened 7 months ago
Hi, to attach a debugger manually to the container, you'll have to provide the debugging arguments yourself.
Using the hello world Python 3.8 template, I added debugpy
to the existing requirements file. Then after sam build
, I ran sam local start-api -d 9999 --debug --debug-args "-m debugpy --listen 0.0.0.0:9999"
to start the API.
-m debugpy
, invokes the debugpy
as a Python module--listen 0.0.0.0:9999
, tells debugpy
to look at port 9999
for connectionsAfter waiting for the API to start, I ran curl localhost:3000/hello
and the API waited for me to attach a debugger. Using the following VSCode configuration:
{
"name": "Python: Remote Attach",
"type": "python",
"request": "attach",
"connect": {
"host": "localhost",
"port": 9999
},
"pathMappings": [
{
"localRoot": "${workspaceFolder}/hello_world",
"remoteRoot": "/var/task"
}
],
"justMyCode": false
},
I was able to set a break point and debug the function.
The AWS Toolkit extension abstracts away all this setup as you've shown. Let me know if you aren't able to get the debugger working, or if it still refuses to wait for you to attach a debugging session.
@lucashuy, I will try it out shortly and report back, but can we enhance the sam local start-api
and make these steps (add debugpy and start it) trigger automatically? Similar to how NodeJS seems to handle it. Cheers
@lucashuy, I tried setting the lambda runtime in the SAM template to python3.8 and python 3.11. I then ran sam build
.
For both runtimes, the debugpy module doesn't get triggered. The behavior is that the function executes without pausing for vscode to attach. Inspecting the provisioned container in docker, shows that the arguments didn't propagate
Can you share which version of the sam cli and docker engine you tested with? Thanks
Hi, sorry for missing this, but the --wait-for-client
should be provided as well.
Using SAM CLI 1.107.0
and Docker Engine 24.0.6
I can get the debugger to wait:
sam local start-api -d 9999 --debug --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
{
"Id": "44963bd2dff4803275ece386662aaa0cb80219f9e0d2d0458fa5861e07a5c7e7",
"Created": "2024-01-12T23:39:14.668381755Z",
"Path": "/var/rapid/aws-lambda-rie",
"Args": [
"--log-level",
"error",
"/var/lang/bin/python3.8",
"-m",
"debugpy",
"--listen",
"0.0.0.0:9999",
"--wait-for-client",
"/var/runtime/bootstrap.py"
],
...
...
This is after adding debugpy
as a requirement for the Lambda function.
Hi @lucashuy , this worked with caveats noted below:
launch.json
, you must point pathMappings => localRoot to the app.py folder otherwise the breakpoint won't get hit. Which means that if you have 10 lambda functions you will need 10 definitions in launch.json, does that make sense? With NodeJS the behavior is different, I use 1 remote attach definition in launch.json and it works with all functions regardless of where "localRoot" is pointing to.sam build -t cdk.out/template.yml
to generate .aws-sam
artifacts, otherwise you debugpy won't be copied even if you had a .venvCan we pass the --debug-args transparently when -d 5858
is set and python lambdas are detected?
I could also get the debugger to wait using --debug-args but it wasn't intuitive given other language debuggers don't require this or adding a dependancy.
An added complexity was that I was including my lambda requirements through a aws_cdk.aws_lambda_python_alpha.PythonLayerVersion but adding debugpy here didn't work - I got a module not found error - I suppose /opt/python is added to the path after the debug args are executed. I had to pip install debugpy -t lambda
where lambda is the root of the AssetCode.
@antoineaws I no longer experiencing any of the caveats you mentioned.
SAM CLI 1.113.0 CDK 2.133.0
CDK construct:
getThemes = lambda_.Function( self, "getThemes", runtime=lambda_.Runtime.PYTHON_3_8, code=lambda_.AssetCode.from_asset(path.join(os.getcwd(), "python/lambda")), handler="get_themes.lambda_handler", vpc=vpc, layers=[reqs_layer], tracing=lambda_.Tracing.ACTIVE, timeout=Duration.seconds(45), )
SAM start-api command:
sam local start-api -t ./cdk.out/NameStack-nwbxl.template.json --warm-containers eager --env-vars python/lambda/local.json -d 9999 --debug-function getNameFA47 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
Hi, sorry for missing this, but the
--wait-for-client
should be provided as well.Using SAM CLI
1.107.0
and Docker Engine24.0.6
I can get the debugger to wait:sam local start-api -d 9999 --debug --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
{ "Id": "44963bd2dff4803275ece386662aaa0cb80219f9e0d2d0458fa5861e07a5c7e7", "Created": "2024-01-12T23:39:14.668381755Z", "Path": "/var/rapid/aws-lambda-rie", "Args": [ "--log-level", "error", "/var/lang/bin/python3.8", "-m", "debugpy", "--listen", "0.0.0.0:9999", "--wait-for-client", "/var/runtime/bootstrap.py" ], ... ...
This is after adding
debugpy
as a requirement for the Lambda function.
I tried it, but it outputs an error.
Extract from sam local start-api
log.
START RequestId: ac42975a-b8d7-42df-8c9a-4461e23e8fae Version: $LATEST
14 May 2024 08:22:13,104 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: killed InvokeID=
14 May 2024 08:22:13,116 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: killed InvokeID=f7e10428-774e-4580-b2f5-206259b0 f4a8
14 May 2024 08:22:13,134 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-14 17:22:13,082 | Failed to deserialize response from RIE, returning the raw response as is
2024-05-14 17:22:14,091 | Invalid lambda response received: Lambda response must be valid json
2024-05-14 17:22:14,093 | Lambda execution failed ()
2024-05-14 17:22:14 127.0.0.1 - - [14/May/2024 17:22:14] "GET /hello HTTP/1.1" 502 -
sam --version SAM CLI, version 1.116.0
@antoineaws
Can you tell me the launch.json and sam local
command for the success?
I am aware that the debugger mode is probably not ready to wait in the first place...
https://github.com/aws/aws-sam-cli/assets/123167609/028a43d4-2527-4f5c-ac4e-0fcba0a96f17
@skurihara-cw The sam command is similar to what @lucashuy suggested here: https://github.com/aws/aws-sam-cli/issues/6546#issuecomment-1890148136 Here's my launch.json: (note direct invoke and attach)
{
"version": "0.2.0",
"configurations": [
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "Python: Function 1 + Payload",
"invokeTarget": {
"target": "code",
"projectRoot": "${workspaceFolder}/xxx/functions/python-function-1",
"lambdaHandler": "app.lambda_handler"
},
"lambda": {
"runtime": "python3.9",
"payload": {
"path": "${workspaceFolder}/xxx/functions/events/apigw-event.json"
},
"environmentVariables": {}
},
"aws": {
"credentials": "profile:my_profile",
"region": "ca-central-1"
}
},
{
"name": "Python: Remote Attach",
"type": "python",
"request": "attach",
"host": "127.0.0.1",
"port": 5858,
"pathMappings": [
{
"localRoot": "${workspaceFolder}/xxxx/functions/python-function-2",
"remoteRoot": "/var/task"
}
],
"justMyCode": true
}
]
}
@lucashuy Bravo for spelling out in detail how to make this work!!! So many "solutions" out there lacking critical details.
@bettlebrox Thank you as well for spelling out in detail how to make it work for CDK, as well as that it doesn't require a "sam build" every time you change code.
For any other rookies (like me) who followed the CDK serverless tutorial in the docs at https://docs.aws.amazon.com/cdk/v2/guide/serverless_example.html, this is everything I had to do beyond what the tutorial tells you to do in order to get remote debugging going for python. First, my final directory structure for context:
You also need to have Docker installed, and follow the instructions at https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html to install the AWS SAM CLI. The other docs in that SAM guide are a helpful reference.
The CDK serverless example tutorial gives you a JS version of the HelloWorld lambda function, but you want python. In the top of your project, create the hello_lambda
directory you see in my project. Create an empty __init__.py
under that (as shown), then create app.py
(highlighted in the screenshot) with these contents (the silly return_message stuff is to have some lines to test debugging):
import json
def lambda_handler(event, context):
return_message = "hello"
return_message += " world!"
return {
"statusCode": 200,
"body": json.dumps({
"message": return_message,
}),
}
To finish switching from Node.JS to Python for the lambda function, edit the cdk_hello_world_stack.py
file and replace the hello_world_function
declaration that defines the Node.JS version of hello world to this:
hello_world_function = aws_lambda.Function( self, "HelloWorldFunction", runtime=aws_lambda.Runtime.PYTHON_3_9, handler='app.lambda_handler', code=aws_lambda.Code.from_asset('./hello_lambda') )
Later for debugging, you'll need the debugpy module in this same directory (seems like there's a bug that SAM doesn't put it in the target Docker container for you). As @bettlebrox said, do it like this python3 -m pip install debugpy -t hello_lambda
- as my screenshot showed, it should be right there in your hello_lambda
directory.
Next (still at the top level in the command line), run cdk synth --no-staging
. Now you need the unique name it assigned to the function (it might be called the logical id, but I'm a noob so I don't know). You can find this in the mess it spits out to the terminal in a section like this:
HelloWorldFunctionB2AB6E79: Type: AWS::Lambda::Function
In my case it wasn't far from the top. You can also find it in cdk.out/CdkHelloWorldStack.template.json
, for me it was the second object under Resources
and looked like this:
"HelloWorldFunctionB2AB6E79": { "Type": "AWS::Lambda::Function",
In either case, you take that name and put it into the command that @bettlebrox gave us, like this:
sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
At this point, your server should be started and waiting for you to send it HTTP requests. Give it a try with a second terminal window like this curl http://127.0.0.1:3000/hello
. It should sit there and not respond. It's waiting for you to connect a debugger. If it errors, or returns something right away- I'm sorry but something has gone wrong, fix it before you proceed.
Create a .vscode/launch.json file, and give it the following contents (or edit your existing one to include this configuration):
{ // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: Remote Attach hello lambda", "type": "debugpy", "request": "attach", "connect": { "host": "localhost", "port": 9999 }, "pathMappings": [ { "localRoot": "${workspaceFolder}/hello_lambda", "remoteRoot": "/var/task" } ], "justMyCode": false } ] }
Set a breakpoint in the app.py
on one of those return_message
lines. In your "Run and Debug" tab in VSCode, run this debugger. If all is correct, your debugger should break immediately on your breakpoint. Continuing should let the function finish and you should see the response in your second terminal window. You can now edit the app.py
file, save it, and issue a new HTTP request without doing anything else at the command line. If you edit the file, you will have to relaunch the debugger (it's because the docker container restarts with the new file contents so disconnects the debugger). If you don't edit the file and just want to debug again, it'll just work.
@jpangburn
sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
Are the following options always necessary? Can they be supplemented?🥺
--warm-containers eager
--debug-function HelloWorldFunctionB2AB6E79
@jpangburn
sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
Are the following options always necessary? Can they be supplemented?🥺
--warm-containers eager
--debug-function HelloWorldFunctionB2AB6E79
@skurihara-cw I wondered the same thing myself and tested them but felt like my previous comment was too long already, so left it out.
For eager/lazy, I noticed after you run/edit the function it always says Lambda Function 'HelloWorldFunction' source code has been changed, terminate its warm container. The new container will be created in lazy mode
. So after the first run it definitely doesn't matter because then it's lazy no matter what you said. On the first run though, in eager mode if you try to connect your debugger before you make the API call you don't get an error message- just the debugger doesn't connect, but in lazy mode you get an error message you have to close (connection refused). In all cases, you can't connect the debugger after the API call until a few seconds have passed and the container accepts the connection. So I just left it in case @bettlebrox knew something I didn't. I suspect in other versions of SAM it might be necessary because the documentation says it's tied to the --debug-function
parameter. Again, I'm a noob here, I started with CDK yesterday, and tried SAM once some years ago so take everything I say with that in mind.
The --debug-function
parameter is, unfortunately, critical. When I tested it, without that function the API call completes without waiting and you cannot connect the debugger. It appears tied to the --debug-port
aka -d
and --debug-args
options. I think what's happening is that each lambda function in your CDK is deployed to a docker container and only one container can be debugging at a time (obviously because you can only open the same port once), so I think you need that --debug-function to tell SAM which container to open the debug port on? That's a guess though.
Why you need the -d 9999
and also having --listen 0.0.0.0:9999
in the --debug-args
, I don't know and didn't try to remove it. This thing was so hard to get debugging, mostly I was just relieved it finally worked 😅 so I only tried the above two things you happened to ask about. I tried it now out of curiosity, and removing either of those switches with 9999
did not work. I suspect one is for SAM to listen locally (the former one) and the latter one is for debugpy inside the container to listen on (SAM must connect to that container somehow and make the local port available to connect to from VSCode). In any case, they are required.
@jpangburn
sam local start-api -t ./cdk.out/CdkHelloWorldStack.template.json --warm-containers eager -d 9999 --debug-function HelloWorldFunctionB2AB6E79 --debug-args "-m debugpy --listen 0.0.0.0:9999 --wait-for-client"
Are the following options always necessary? Can they be supplemented?🥺
--warm-containers eager
--debug-function HelloWorldFunctionB2AB6E79
@skurihara-cw I wondered the same thing myself and tested them but felt like my previous comment was too long already, so left it out.
For eager/lazy, I noticed after you run/edit the function it always says
Lambda Function 'HelloWorldFunction' source code has been changed, terminate its warm container. The new container will be created in lazy mode
. So after the first run it definitely doesn't matter because then it's lazy no matter what you said. On the first run though, in eager mode if you try to connect your debugger before you make the API call you don't get an error message- just the debugger doesn't connect, but in lazy mode you get an error message you have to close (connection refused). In all cases, you can't connect the debugger after the API call until a few seconds have passed and the container accepts the connection. So I just left it in case @bettlebrox knew something I didn't. I suspect in other versions of SAM it might be necessary because the documentation says it's tied to the--debug-function
parameter. Again, I'm a noob here, I started with CDK yesterday, and tried SAM once some years ago so take everything I say with that in mind.The
--debug-function
parameter is, unfortunately, critical. When I tested it, without that function the API call completes without waiting and you cannot connect the debugger. It appears tied to the--debug-port
aka-d
and--debug-args
options. I think what's happening is that each lambda function in your CDK is deployed to a docker container and only one container can be debugging at a time (obviously because you can only open the same port once), so I think you need that --debug-function to tell SAM which container to open the debug port on? That's a guess though.Why you need the
-d 9999
and also having--listen 0.0.0.0:9999
in the--debug-args
, I don't know and didn't try to remove it. This thing was so hard to get debugging, mostly I was just relieved it finally worked 😅 so I only tried the above two things you happened to ask about. I tried it now out of curiosity, and removing either of those switches with9999
did not work. I suspect one is for SAM to listen locally (the former one) and the latter one is for debugpy inside the container to listen on (SAM must connect to that container somehow and make the local port available to connect to from VSCode). In any case, they are required.
I had a similar experience tinkering until I got something to work so I just copied in what was working for me. I believe --debug-function is required for debugging but --warm-containers is not. Thats just an assumption I haven't tested it. Also I used 9999 port for debugging because I was debugging something else on the default port so I think both those options can be removed. Again not tested.
Description:
Team, The SAM cli -
sam local start-api --debug-port 5858
doesn't allow developers to debug their code (pause at breakpoints), even if--debug-function HelloWorldFunction
is specified.If you inspect the container that gets launched, note 2 things:
We have the args between python3.11 and bootstrap.py. THe below image is what the args are when using vscode - launch.json - direct invoke. It also seems that the files needed for debugging e.g.
debugpy
.The libraries needed for python debugging are also not copied e.g. debugpy etc. Below is what is copied with aws-sam direct invoke which is the correct files needed to debug:
On the other hand, in the same project, that same command works great with NodeJS.
The expected behavior is to have the SAM lambda container wait on port --debug-port to allow IDE / VSCode to attach to it for debugging. Constantly having to launch direct-invoke tanks the developer experience and sam local start-api is a perfect tool for auto-reload and debugging.
Same with CDK:
Note, the same behavior happens with a CDK project where the command is run against the CDK synth template :
sam local start-api -d 5858 -t ./cdk.out/template.yaml
Steps to reproduce:
sam init
Observed result:
Expected result:
Expected that the sam lambda container will wait using debugpy for the IDE attach to it and then allow a developer to debug.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
sam --version
: 1.106.0Feel free if you need more info. Thanks!