search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.53k stars 1.17k forks source link

"sam local start-api" add data onto payload of POST data, when fwding to lambda. #996

Closed XDanny322 closed 5 years ago

XDanny322 commented 5 years ago

Description

In short - when running sam local start-api, POST some data to the local endpoint so that the data can be sent to a Lambda.. it seems like the data gets "massaged" and are "pushed" under a key called "body".

In long.. I am using sam to create some simple resources. A Lambda Function, and a API Gateway, which i am declaring directly by calling "AWS::Serverless::Api" and not inferring it from an "Event" within a function. Relevant parts of the SAM template below.

...
Resources:
  ApiGatewayToLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: ['sts:AssumeRole']
            Effect: Allow
            Principal:
              Service: ['apigateway.amazonaws.com']
        Version: '2012-10-17'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaRole
        - arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs

  ApiGateway:
    Type: AWS::Serverless::Api
    Properties:
      StageName: test
      EndpointConfiguration: REGIONAL
      DefinitionBody:
        swagger: "2.0"
        info:
          title: "TestAPI"
          description: TestAPI description in Markdown.
        paths:
          /create:
            post:
              x-amazon-apigateway-integration:
                uri:
                  !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyLambda.Arn}/invocations
                credentials: !GetAtt ApiGatewayToLambdaRole.Arn
                responses: {}
                httpMethod: POST
                type: aws
        x-amazon-apigateway-request-validators:
          Validate query string parameters and headers:
            validateRequestParameters: true
            validateRequestBody: false

  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: ['sts:AssumeRole']
            Effect: Allow
            Principal:
              Service: [lambda.amazonaws.com]
        Version: '2012-10-17'
      Path: /
      Policies:
        - PolicyName: CodeBuildAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Action:
              - logs:*
              - lambda:*
              - ec2:CreateNetworkInterface
              - ec2:DescribeNetworkInterfaces
              - ec2:DeleteNetworkInterface
              Effect: Allow
              Resource: "*"
            Version: '2012-10-17'

  MyLambda:
    Type: AWS::Serverless::Function
    Properties:
      Role: !GetAtt LambdaRole.Arn
      Handler: myfunctionname.lambda_handler
      CodeUri: ./src/myfunctionname
      Events:
        SCAPIGateway:
          Type: Api
          Properties:
            RestApiId: !Ref ApiGateway
            Path: /create
            Method: POST
...

When i sam build ....

(venv_sam) [dlai@ip-11.22.33.44 some_working_dir]$ time sam build --use-container --template backend/template.yaml
2019-02-08 21:30:01 Starting Build inside a container
2019-02-08 21:30:02 Found credentials in environment variables.
2019-02-08 21:30:02 Building resource 'MyLambdaFunction'

Fetching lambci/lambda:build-python3.6 Docker container image......
2019-02-08 21:30:02 Mounting /home/dlai/some_working_dir/backend/src/create_orchestrator as /tmp/samcli/source:ro inside runtime container
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

real    0m2.632s
user    0m0.446s
sys     0m0.089s
(venv_sam) [dlai@ip-11.22.33.44 some_working_dir]$

...and sam local start-api all is well. 

(venv_sam) [dlai@ip-11.22.33.44 some_working_dir]$ sam local start-api --host `hostname -I | awk '{print $1}'` --port 3000 --region us-east-1 --template .aws-sam/build/template.yaml --debug
2019-02-08 21:31:38 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-02-08 21:31:38 Changing event name from before-call.apigateway to before-call.api-gateway
2019-02-08 21:31:38 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-02-08 21:31:38 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-02-08 21:31:38 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-02-08 21:31:38 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-02-08 21:31:38 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-02-08 21:31:38 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-02-08 21:31:38 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-02-08 21:31:38 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-02-08 21:31:38 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-02-08 21:31:38 Setting config variable for region to u'us-east-1'
2019-02-08 21:31:38 local start-api command is called
2019-02-08 21:31:38 Looking for credentials via: env
2019-02-08 21:31:38 Found credentials in environment variables.
2019-02-08 21:31:38 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/endpoints.json
2019-02-08 21:31:38 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f02ecbe6c80>
2019-02-08 21:31:38 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/serverlessrepo/2017-09-08/service-2.json
2019-02-08 21:31:38 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x7f02ecc14578>
2019-02-08 21:31:38 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-08 21:31:38 Setting serverlessrepo timeout as (60, 60)
2019-02-08 21:31:38 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/_retry.json
2019-02-08 21:31:38 Registering retry handlers for service: serverlessrepo
2019-02-08 21:31:38 Collected default values for parameters: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2019-02-08 21:31:38 4 resources found in the template
2019-02-08 21:31:38 Found Serverless function with name='MyLambdaFunction' and CodeUri='MyLambdaFunction'
2019-02-08 21:31:38 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:38 No config file found
2019-02-08 21:31:38 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:38 No config file found
2019-02-08 21:31:38 http://localhost:None "GET /v1.35/_ping HTTP/1.1" 200 2
2019-02-08 21:31:38 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f02ecbe6c80>
2019-02-08 21:31:38 Event creating-client-class.serverlessapplicationrepository: calling handler <function add_generate_presigned_url at 0x7f02ecc14578>
2019-02-08 21:31:38 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-08 21:31:38 Setting serverlessrepo timeout as (60, 60)
2019-02-08 21:31:38 Registering retry handlers for service: serverlessrepo
2019-02-08 21:31:39 Collected default values for parameters: xxxxxxxxxxxxxx
2019-02-08 21:31:39 4 resources found in the template
2019-02-08 21:31:39 Found '1' API Events in Serverless function with name 'MyLambdaFunction'
2019-02-08 21:31:39 Detected Inline Swagger definition
2019-02-08 21:31:39 Lambda function integration not found in Swagger document at path='/create' method='post'
2019-02-08 21:31:39 Found '0' APIs in resource 'ApiGatewaySCBackend'
2019-02-08 21:31:39 Removed duplicates from '1' Explicit APIs and '0' Implicit APIs to produce '1' APIs
2019-02-08 21:31:39 1 APIs found in the template
2019-02-08 21:31:39 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f02ecbe6c80>
2019-02-08 21:31:39 Loading JSON file: /home/dlai/venv_sam/lib/python2.7/site-packages/botocore/data/lambda/2015-03-31/service-2.json
2019-02-08 21:31:39 Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x7f02ecc14578>
2019-02-08 21:31:39 The s3 config key is not a dictionary type, ignoring its value of: None
2019-02-08 21:31:39 Setting lambda timeout as (60, 60)
2019-02-08 21:31:39 Registering retry handlers for service: lambda
2019-02-08 21:31:39 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:39 No config file found
2019-02-08 21:31:39 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:31:39 No config file found
2019-02-08 21:31:39 Mounting MyLambdaFunction at http://11.22.33.44:3000/create [POST]
2019-02-08 21:31:39 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2019-02-08 21:31:39 Localhost server is starting up. Multi-threading = True
2019-02-08 21:31:39  * Running on http://11.22.33.44:3000/ (Press CTRL+C to quit)

However, when i curl post to it and send in some data, it seems like the APIGateway is massaging the data.

The POST, note how the data is only one key value pair of {"id": "61763eb7-c980-4ef1-a58d-04bd8b8caf14"}

[dlai@ip-11.22.33.44 ~]$ curl -d '{"id": "61763eb7-c980-4ef1-a58d-04bd8b8caf14"}' -H "Content-Type: application/json" -X POST http://11.22.33.44:3000/create
no data[dlai@ip-11.22.33.44 ~]$

sam local's APIGatway response:

2019-02-08 21:33:30 Constructed String representation of Event to invoke Lambda. Event: {"body": "{\"id\": \"61763eb7-c980-4ef1-a58d-04bd8b8caf14\"}", "httpMethod": "POST", "resource": "/create", "queryStringParameters": null, "requestContext": {"httpMethod": "POST", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "path": "/create", "extendedRequestId": null, "resourceId": "123456", "apiId": "1234567890", "stage": "prod", "resourcePath": "/create", "identity": {"accountId": null, "apiKey": null, "userArn": null, "cognitoAuthenticationProvider": null, "cognitoIdentityPoolId": null, "userAgent": "Custom User Agent String", "caller": null, "cognitoAuthenticationType": null, "sourceIp": "11.22.33.44", "user": null}, "accountId": "123456789012"}, "headers": {"Content-Length": "46", "X-Forwarded-Proto": "http", "X-Forwarded-Port": "3000", "Content-Type": "application/json", "Host": "11.22.33.44:3000", "Accept": "*/*", "User-Agent": "curl/7.29.0"}, "stageVariables": null, "path": "/create", "pathParameters": null, "isBase64Encoded": false}
2019-02-08 21:33:30 Found one Lambda function with name 'MyLambdaFunction'
2019-02-08 21:33:30 Invoking create_orchestrator.lambda_handler (python3.6)
2019-02-08 21:33:30 Environment variables overrides data is standard format
2019-02-08 21:33:30 Loading AWS credentials from session with profile 'default'
2019-02-08 21:33:30 Resolving code path. Cwd=/home/dlai/some_working_dir/.aws-sam/build, CodeUri=MyLambdaFunction
2019-02-08 21:33:30 Resolved absolute path to code is /home/dlai/some_working_dir/.aws-sam/build/MyLambdaFunction
2019-02-08 21:33:30 Code /home/dlai/some_working_dir/.aws-sam/build/MyLambdaFunction is not a zip/jar file
2019-02-08 21:33:30 Skipping building an image since no layers were defined
2019-02-08 21:33:30 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:33:30 No config file found
2019-02-08 21:33:30 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:33:30 No config file found
2019-02-08 21:33:30 http://localhost:None "GET /v1.35/images/lambci/lambda:python3.6/json HTTP/1.1" 200 None
2019-02-08 21:33:30 Looking for auth config
2019-02-08 21:33:30 No auth config in memory - loading from filesystem
2019-02-08 21:33:30 Trying paths: ['/home/dlai/.docker/config.json', '/home/dlai/.dockercfg']
2019-02-08 21:33:30 No config file found
2019-02-08 21:33:30 Looking for auth entry for 'docker.io'
2019-02-08 21:33:30 No entry found
2019-02-08 21:33:30 No auth config found
2019-02-08 21:33:30 http://localhost:None "POST /v1.35/images/create?tag=python3.6&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:python3.6 Docker container image......
2019-02-08 21:33:30 Mounting /home/dlai/some_working_dir/.aws-sam/build/MyLambdaFunction as /var/task:ro inside runtime container
2019-02-08 21:33:30 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2019-02-08 21:33:30 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:30 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:31 http://localhost:None "POST /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/start HTTP/1.1" 204 0
2019-02-08 21:33:31 Starting a timer for 180 seconds for function 'MyLambdaFunction'
2019-02-08 21:33:31 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:31 http://localhost:None "POST /containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/attach?stream=1&stdin=0&logs=1&stderr=1&stdout=1 HTTP/1.1" 101 0
START RequestId: 4043c553-31ac-45dd-910a-0951478b346f Version: $LATEST
Version: 0.63
Processing request.
~~~~~~~~~~~~~~
The event box looks like this:
~~~~~~~~~~~~~~
{"body": "{\"id\": \"61763eb7-c980-4ef1-a58d-04bd8b8caf14\"}", "httpMethod": "POST", "resource": "/create", "queryStringParameters": null, "requestContext": {"httpMethod": "POST", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "path": "/create", "extendedRequestId": null, "resourceId": "123456", "apiId": "1234567890", "stage": "prod", "resourcePath": "/create", "identity": {"accountId": null, "apiKey": null, "userArn": null, "cognitoAuthenticationProvider": null, "cognitoIdentityPoolId": null, "userAgent": "Custom User Agent String", "caller": null, "cognitoAuthenticationType": null, "sourceIp": "11.22.33.44", "user": null}, "accountId": "123456789012"}, "headers": {"Content-Length": "46", "X-Forwarded-Proto": "http", "X-Forwarded-Port": "3000", "Content-Type": "application/json", "Host": "11.22.33.44:3000", "Accept": "*/*", "User-Agent": "curl/7.29.0"}, "stageVariables": null, "path": "/create", "pathParameters": null, "isBase64Encoded": false}
'id': KeyError
Traceback (most recent call last):
  File "/var/task/create_orchestrator.py", line 660, in lambda_handler
    result = _process_request(event)
  File "/var/task/create_orchestrator.py", line 625, in _process_request
    request_id = event['id']
KeyError: 'id'

END RequestId: 4043c553-31ac-45dd-910a-0951478b346f
REPORT RequestId: 4043c553-31ac-45dd-910a-0951478b346f Duration: 291 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 28 MB
2019-02-08 21:33:31 http://localhost:None "GET /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0/json HTTP/1.1" 200 None
2019-02-08 21:33:31 http://localhost:None "DELETE /v1.35/containers/8d117ae516a342a817159691db2ba4944746d643a7cc57cda6e5daaa9eaf9fd0?force=True&link=False&v=False HTTP/1.1" 204 0
2019-02-08 21:33:31 No Content-Type given. Defaulting to 'application/json'.
2019-02-08 21:33:31 11.22.33.44 - - [08/Feb/2019 21:33:31] "POST /create HTTP/1.1" 200 -

I have tested the post from curl, and postman, and its the same; that it causes this problem. I have also tested against a real APIGatway running in AWS, and the problem does not happen.

The point here is, when i am posting this data:

{
    "id": "61763eb7-c980-4ef1-a58d-04bd8b8caf14"
}

sam local APIGateway seems to massage the data, puts MY data under a key called "body".. add other header, etc stuff, and then pass it to my lambda.

{
    "body": "{\"id\": \"61763eb7-c980-4ef1-a58d-04bd8b8caf14\"}",
    "headers": {
        "Accept": "*/*",
        "Content-Length": "46",
        "Content-Type": "application/json",
        "Host": "11.22.33.44:3000",
        "User-Agent": "curl/7.29.0",
        "X-Forwarded-Port": "3000",
        "X-Forwarded-Proto": "http"
    },
    "httpMethod": "POST",
    "isBase64Encoded": false,
    "path": "/create",
    "pathParameters": null,
    "queryStringParameters": null,
    "requestContext": {
        "accountId": "123456789012",
        "apiId": "1234567890",
        "extendedRequestId": null,
        "httpMethod": "POST",
        "identity": {
            "accountId": null,
            "apiKey": null,
            "caller": null,
            "cognitoAuthenticationProvider": null,
            "cognitoAuthenticationType": null,
            "cognitoIdentityPoolId": null,
            "sourceIp": "11.22.33.44",
            "user": null,
            "userAgent": "Custom User Agent String",
            "userArn": null
        },
        "path": "/create",
        "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
        "resourceId": "123456",
        "resourcePath": "/create",
        "stage": "prod"
    },
    "resource": "/create",
    "stageVariables": null
}

Steps to reproduce

See description

Observed result

sam local start-api seems to massage data before sending to lambda.

Expected result

The data i post from curl / postman, gets sent as is, to the lambda.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. Testing on RHEL 7.6 EC2
  2. Testing on python pip, aws-sam-cli, version 0.7.0 to 0.11.0. Complete pip freeze below
(venv_sam) [dlai@someserver]$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
arrow==0.13.0
aws-lambda-builders==0.0.5
aws-sam-cli==0.11.0
aws-sam-translator==1.9.0
awscli==1.16.96
awslogs==0.11.0
backports.functools-lru-cache==1.5
backports.ssl-match-hostname==3.7.0.1
binaryornot==0.4.4
boto3==1.9.86
botocore==1.12.86
certifi==2018.11.29
chardet==3.0.4
chevron==0.13.1
click==6.7
colorama==0.3.9
cookiecutter==1.6.0
dateparser==0.7.0
docker==3.7.0
docker-pycreds==0.4.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
functools32==3.2.3.post2
future==0.17.1
futures==3.2.0
idna==2.7
ipaddress==1.0.22
itsdangerous==1.1.0
Jinja2==2.10
jinja2-time==0.2.0
jmespath==0.9.3
jsonschema==2.6.0
MarkupSafe==1.1.0
pathlib2==2.3.3
pathspec==0.5.9
poyo==0.4.2
pyasn1==0.4.5
python-dateutil==2.7.5
pytz==2018.9
PyYAML==3.13
regex==2019.1.24
requests==2.20.1
rsa==3.4.2
s3transfer==0.1.13
scandir==1.9.0
serverlessrepo==0.1.5
six==1.11.0
termcolor==1.1.0
tzlocal==1.5.1
urllib3==1.24.1
websocket-client==0.54.0
Werkzeug==0.14.1
whichcraft==0.5.2
yamllint==1.14.0
XDanny322 commented 5 years ago

Sorry, i meant to create this issue in this repo: https://github.com/awslabs/serverless-application-model

Closing this issue.

Leooo commented 3 years ago

having the same problem, shame I can't find the equivalent issue in https://github.com/awslabs/serverless-application-model @XDanny322

davidjb commented 1 year ago

This issue appears to be by design - sam local start-api starts up a local API gateway and HTTP requests get constructed into API Gateway Lambda Events (https://github.com/aws/aws-sam-cli/blob/develop/samcli/local/apigw/event_constructor.py), which as @XDanny322 noted, takes the HTTP request body (amongst other fields) and puts that into the corresponding body field on the event. Coming at this from a non-API perspective, this wasn't obvious behaviour and not something that seems documented - all the start-api docs I've seen refer to GET requests without a request body.

In short, any lambda functions that are configured for API Events are expected to receive API Gateway Events, which makes sense in hindsight. This means that in order to call such a function otherwise (e.g. with sam local invoke) you would need to create the expected gateway event structure. Or, conversely, if a given function isn't intended for API event handling, then skip using sam local start-api, remove the Events handler, and just use sam local invoke to directly call the function instead.