aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.5k stars 1.17k forks source link

Bug: Inconsistent behavior when using HTTP API in local environment vs Lambda #5860

Open leandrodamascena opened 1 year ago

leandrodamascena commented 1 year ago

Description:

Hello everybody! Following @moelasmar's recommendation in issue #5579, I am opening a new issue with more information about the case. Along with Ruben, I'm building Powertools for AWS Lambda and a customer has raised this issue: aws-powertools/powertools-lambda-python#2765.

I'll go into more detail throughout this issue, but to summarize: the v2.0 payload of an HTTP API (AWS::Serverless::HttpApi) behaves differently when using sam local start-api and when running in the API Gateway + Lambda environment on AWS.

In order not to focus on a specific tool, I will not consider the use of Powertools here, but only payloads. I think it will be easier to understand if this is expected behavior from the SAM CLI or a possible bug.

Steps to reproduce:

I'm using the following SAM template with a specific StageName for HttpApi:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    sam-app

Globals: 
    Function:
        Timeout: 5
        MemorySize: 128
        Runtime: python3.10

Resources:
    HttpApi:
        Type: AWS::Serverless::HttpApi
        Properties:
            StageName: prod

    HelloWorldFunction:
        Type: AWS::Serverless::Function
        Properties:
            Handler: app.lambda_handler
            CodeUri: hello_world
            Description: Hello World function
            Architectures:
                - x86_64
            Tracing: Active
            Events:
                HelloPath:
                    Type: HttpApi
                    Properties:
                        ApiId: !Ref HttpApi
                        Path: /hello
                        Method: GET

And I have the following Lambda code:

import json

def lambda_handler(event: dict, context) -> dict:
    print(event)

The requirements.txt is the basic with only requests library and I'm running sam build and sam local start-api

❯ sam local start-api
Initializing the lambda functions containers.                                                                                                                                                                                                                                                                            
Local image is up-to-date                                                                                                                                                                                                                                                                                                
Using local image: public.ecr.aws/lambda/python:3.10-rapid-x86_64.                                                                                                                                                                                                                                                       

Mounting /home/leandro/DEVEL-PYTHON/tmp/sam-api-httpapi/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                                                                                                                                                                            
Containers Initialization is done.                                                                                                                                                                                                                                                                                       
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]                                                                                                                                                                                                                                                         
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. If you used sam build before running local commands, you will need to re-run sam build for the changes to be picked up.
You only need to restart SAM CLI if you update your AWS SAM template                                                                                                                                                                                                                                                     
2023-08-28 23:40:40 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000

After that, I invoked the URL http://127.0.0.1:3000/hello just to print the payload I get in the Lambda function.

Observed result:

When running it locally, I see that rawPath and path are just /hello, it's not adding the stage name. When I run this in the API Gateway + Lambda environment I see that the rawPath and path are /prod/hello, that is, the stage name is added.

Below are the payloads received in both environments. An important point to report here is that in API Gateway I deployed this stack using the default APIGateway URL (....execute-api.us-east-1.amazonaws.com) and with a custom domain. In both scenarios the behavior are the same: the stage is added to the rawPath and path.

Payload using SAM CLI

{
  "body": "",
  "cookies": [],
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Accept-Encoding": "deflate, gzip",
    "Accept-Language": "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7",
    "Cache-Control": "max-age=0",
    "Connection": "keep-alive",
    "Host": "127.0.0.1:3000",
    "Sec-Ch-Ua": "\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": "\"Linux\"",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "X-Forwarded-Port": "3000",
    "X-Forwarded-Proto": "http"
  },
  "isBase64Encoded": false,
  "pathParameters": {},
  "rawPath": "/hello",
  "rawQueryString": "",
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "1234567890",
    "domainName": "localhost",
    "domainPrefix": "localhost",
    "http": {
      "method": "GET",
      "path": "/hello",
      "protocol": "HTTP/1.1",
      "sourceIp": "127.0.0.1",
      "userAgent": "Custom User Agent String"
    },
    "requestId": "65df69cd-ac5d-4f41-8257-126097e297e1",
    "routeKey": "GET /hello",
    "stage": "prod",
    "time": "28/Aug/2023:17:08:18 +0000",
    "timeEpoch": 1693242498
  },
  "routeKey": "GET /hello",
  "stageVariables": null,
  "version": "2.0"
}

Payload using API Gateway + Lambda

{
    "version":"2.0",
    "routeKey":"GET /hello",
    "rawPath":"/prod/hello",
    "rawQueryString":"",
    "headers":{
       "accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
       "accept-encoding":"deflate, gzip",
       "accept-language":"pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7",
       "content-length":"0",
       "host":"--REDACTED--",
       "sec-ch-ua":"\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
       "sec-ch-ua-mobile":"?0",
       "sec-ch-ua-platform":"\"Linux\"",
       "sec-fetch-dest":"document",
       "sec-fetch-mode":"navigate",
       "sec-fetch-site":"none",
       "sec-fetch-user":"?1",
       "upgrade-insecure-requests":"1",
       "user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
       "x-amzn-trace-id":"Root=1-64ed1d2b-321767c34bdc22f96d2d3798",
       "x-forwarded-for":"--REDACTED--",
       "x-forwarded-port":"443",
       "x-forwarded-proto":"https"
    },
    "requestContext":{
       "accountId":"--REDACTED--",
       "apiId":"--REDACTED--",
       "domainName":"--REDACTED--.cloud",
       "domainPrefix":"testapigw",
       "http":{
          "method":"GET",
          "path":"/prod/hello",
          "protocol":"HTTP/1.1",
          "sourceIp":"--REDACTED--",
          "userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
       },
       "requestId":"KZF-2gADoAMEPKg=",
       "routeKey":"GET /hello",
       "stage":"prod",
       "time":"28/Aug/2023:22:18:19 +0000",
       "timeEpoch":1693261099547
    },
    "isBase64Encoded":false
 }

Expected result:

When running in a local environment with sam local start-api it is expected to behave the same as when running in an AWS environment. The user can rely on the rawPath and path fields to make decisions such as which internal function to call, which route to map (in micro-frameworks, for example), among other things. I don't know if this is the expected behavior of the SAM CLI, but I think the experience should be the same when running it locally.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

{
  "version": "1.95.0",
  "system": {
    "python": "3.11.3",
    "os": "Linux-5.14.10-300.fc35.x86_64-x86_64-with-glibc2.35"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.16",
    "aws_cdk": "2.77.0 (build 06a0b19)",
    "terraform": "1.1.9"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Thank you very much for your attention. I hope that we can clarify this things and perhaps improve the user experience even more.

hawflau commented 1 year ago

@leandrodamascena thanks for raising the issue. Marking it as a bug and will prioritize on fixing it.

leandrodamascena commented 1 year ago

@leandrodamascena thanks for raising the issue. Marking it as a bug and will prioritize on fixing it.

Thank you @hawflau! I will let our customer know the SAM project is fixing this bug! 💯

mndeveci commented 1 year ago

Hi @leandrodamascena

As we discussed offline, we need to de-prioritize it for now. At high level;

  1. The rawPath variable inside the function payload is actually generated from the mapping that sam local start-api creates, which is missing the stage information from API resource. Changing just event payload will not resolve this problem, in fact it might make it worse given that some customers might already adopted current behavior.
  2. Adding stage name to the mounting path will also make it tricky for existing customers since this would change default behavior. We could mount the same function to more paths (like /stage/path and /path) but this will introduce more changes to the existing functionality since SAM CLI don't parse staging information as of now (ex; there might be multiple stages for an API resource, which is not available at the moment).

Thanks!

robbie-cahill commented 5 months ago

Also just got tripped up by the different case type for headers in SAM local vs AWS (for example, on local, the host header is Host in AWS, the host header is host). Now I must account for both scenarios and have local specific code deployed to production to support each form.

I can see this has been marked as a feature, not a bug. If SAM is intended to provide a way to run Lambda functions locally, any differences in the local environment compared to in AWS are clearly a bug. This is basic stuff for running anything locally, having a realistic production like environment is a must have, not a nice to have.

I should point out here that Serverless Framework does not have this issue (or a bunch of other issues I've encountered with SAM, like no easy local debugging of my TypeScript files, not picking up changes for hot reload, needing to do a "cold start" for every change).

Hatter1337 commented 2 months ago

I have related bug. I'm using AWS SAM with Powertools for AWS Lambda .

Using SAM, I created an HTTP API and a Lambda function with event on GET /health request:

  # ...
  AwesomeApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      Name: !Sub "awesome-api-${Env}"
      StageName: !If [IsProductionEnv, "api", !Sub "${Env}-api"]
      DefaultRouteSettings:
        DetailedMetricsEnabled: true
        ThrottlingBurstLimit: 100
        ThrottlingRateLimit: 100
   # ...
   HealthLambdaFn:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub "health-lambda-fn-${Env}"
      CodeUri: src/lambda/functions/api_users/
      Handler: handler.lambda_handler
      Role: !GetAtt BasicLambdaRole.Arn
      Events:
        CheckHealth:
          Type: HttpApi
          Properties:
            ApiId: !Ref AwesomeApi
            Path: /health
            Method: GET
            TimeoutInMillis: 12000
            PayloadFormatVersion: "2.0"

According to the Powertools documentation, I use APIGatewayHttpResolver:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.event_handler import APIGatewayHttpResolver
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger()
app = APIGatewayHttpResolver()

@app.get("/health")
def check_health():
    return {"status": "healthy"}

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

And it works when I deploy it to AWS. But when running locally via sam local start-api I get 404 because locally the event is different and route unmatched.

To fix this, I have to change the event on the fly:

@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    # Fix for SAM local
    if environ.get("AWS_SAM_LOCAL"):
        event["rawPath"] = "/dev-api" + event["rawPath"]
        event["requestContext"]["http"]["path"] = "/dev-api" + event["requestContext"]["http"]["path"]

    return app.resolve(event, context)