search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.5k stars 1.17k forks source link

Issues on SAM 1.79.0 fails locally after deployed at AWS - No response from invoke container: Lambda response must be valid json #5007

Closed felipeoriani closed 1 year ago

felipeoriani commented 1 year ago

Description:

Same issues here. I don't know why it started happening. I was working in my code and running it using sam local start-api and suddenlly it started breaking with:

Invalid lambda response received: Lambda response must be valid json

It started happening after I deployed at AWS with sam deploy - I don't know if it breaked because of this.

Steps to reproduce:

Observed result:

# sam local start-api 
...
Lambda function 'HelloWorldFunction' is already running
Starting a timer for 3 seconds for function 'HelloWorldFunction'
Function 'HelloWorldFunction' timed out after 3 seconds
No response from invoke container for HelloWorldFunction
Invalid lambda response received: Lambda response must be valid json
2023-04-12 19:04:28 127.0.0.1 - - [12/Apr/2023 19:04:28] "GET /hello HTTP/1.1" 502 -

Expected result:

Should be able to see the json response from HelloWorldFunction from the SAM Template in browser or http client (postman).

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Windows 11
  2. SAM CLI, version 1.79.0
  3. aws-cli/2.11.11 Python/3.11.2 Windows/10 exe/AMD64 prompt/off
{
  "version": "1.79.0",
  "system": {
    "python": "3.8.8",
    "os": "Windows-10-10.0.22621-SP0"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.22",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  }
}

Add --debug flag to command you are running

# sam local start-api --debug
2023-04-12 19:04:17,245 | Config file location: D:\Projects\some-app\samconfig.toml
2023-04-12 19:04:17,245 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in config file at 'D:\Projects\some-app\samconfig.toml'...
2023-04-12 19:04:17,728 | Create Observer for resource D:\Projects\some-app\.aws-sam\build with recursive False
2023-04-12 19:04:17,738 | Initializing the lambda functions containers.
2023-04-12 19:04:17,740 | Async execution started
2023-04-12 19:04:17,740 | Invoking function functools.partial(<function InvokeContext._initialize_all_functions_containers.<locals>.initialize_function_container at 0x00000222D2BA7550>, Function(function_id='HelloWorldFunction', name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='nodejs18.x', memory=None, timeout=3, handler='app.lambdaHandler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='D:\\Projects\\some-app\\.aws-sam\\build\\HelloWorldFunction', environment={'Variables': {'NODE_OPTIONS': ' --enable-source-maps'}}, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'BuildMethod': 'esbuild', 'BuildProperties': {'EntryPoints': ['app.ts'], 'Minify': True, 'Sourcemap': True, 'Target': 'es2020'}, 'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], function_url_config=None, stack_path='', runtime_management_config=None))2023-04-12 19:04:17,741 | Waiting for async results
2023-04-12 19:04:17,747 | Loading AWS credentials from session with profile 'None'
2023-04-12 19:04:17,755 | Resolving code path. Cwd=D:\Projects\some-app\.aws-sam\build, CodeUri=D:\Projects\some-app\.aws-sam\build\HelloWorldFunction
2023-04-12 19:04:17,755 | Resolved absolute path to code is D:\Projects\some-app\.aws-sam\build\HelloWorldFunction
2023-04-12 19:04:17,757 | watch resource D:\Projects\some-app\.aws-sam\build\HelloWorldFunction
2023-04-12 19:04:17,757 | Create Observer for resource D:\Projects\some-app\.aws-sam\build\HelloWorldFunction with recursive True
2023-04-12 19:04:17,758 | watch resource D:\Projects\some-app\.aws-sam\build\HelloWorldFunction's parent D:\Projects\some-app\.aws-sam\build
2023-04-12 19:04:17,759 | Code D:\Projects\some-app\.aws-sam\build\HelloWorldFunction is not a zip/jar file
2023-04-12 19:04:20,941 | Local image is up-to-date
2023-04-12 19:04:20,952 | Using local image: public.ecr.aws/lambda/nodejs:18-rapid-x86_64.

2023-04-12 19:04:20,952 | Mounting D:\Projects\some-app\.aws-sam\build\HelloWorldFunction as /var/task:ro,delegated, inside runtime container
2023-04-12 19:04:21,488 | Async execution completed
2023-04-12 19:04:21,489 | Containers Initialization is done.
2023-04-12 19:04:21,489 | Found '1' API Events in Serverless function with name 'HelloWorldFunction'
2023-04-12 19:04:21,489 | Detected Inline Swagger definition
2023-04-12 19:04:21,489 | Lambda function integration not found in Swagger document at path='/hello' method='get'
2023-04-12 19:04:21,489 | Found '0' APIs in resource 'ServerlessRestApi'
2023-04-12 19:04:21,489 | Removed duplicates from '0' Explicit APIs and '1' Implicit APIs to produce '1' APIs
2023-04-12 19:04:21,489 | 1 APIs found in the template
2023-04-12 19:04:21,490 | Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
2023-04-12 19:04:21,491 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. If you used sam build before running local commands, you will need to re-run sam build for the changes to be picked up. You only need to restart SAM CLI if you update your AWS SAM template
2023-04-12 19:04:21,491 | Localhost server is starting up. Multi-threading = True
2023-04-12 19:04:21 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
2023-04-12 19:04:21 Press CTRL+C to quit
2023-04-12 19:04:24,662 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate, br", "Connection": "keep-alive", "Host": "127.0.0.1:3000", "Postman-Token": "e0be7d5d-5fa5-4f59-88a5-c826fab4edd4", "User-Agent": "PostmanRuntime/7.32.2", "X-Forwarded-Port": "3000", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["*/*"], "Accept-Encoding": ["gzip, deflate, br"], "Connection": ["keep-alive"], "Host": ["127.0.0.1:3000"], "Postman-Token": ["e0be7d5d-5fa5-4f59-88a5-c826fab4edd4"], "User-Agent": ["PostmanRuntime/7.32.2"], "X-Forwarded-Port": ["3000"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/hello", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "127.0.0.1:3000", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "127.0.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/hello", "protocol": "HTTP/1.1", "requestId": "110a774a-e9fc-49bb-b94e-3f1baab320aa", "requestTime": "12/Apr/2023:22:04:17 +0000", "requestTimeEpoch": 1681337057, "resourceId": "123456", "resourcePath": "/hello", "stage": "Prod"}, "resource": "/hello", "stageVariables": null, "version": "1.0"}
2023-04-12 19:04:24,662 | Found one Lambda function with name 'HelloWorldFunction'
2023-04-12 19:04:24,662 | Invoking app.lambdaHandler (nodejs18.x)
2023-04-12 19:04:24,663 | Resolving code path. Cwd=D:\Projects\some-app\.aws-sam\build, CodeUri=D:\Projects\some-app\.aws-sam\build\HelloWorldFunction
2023-04-12 19:04:24,663 | Resolved absolute path to code is D:\Projects\some-app\.aws-sam\build\HelloWorldFunction
2023-04-12 19:04:24,670 | Reuse the created warm container for Lambda function 'HelloWorldFunction'
2023-04-12 19:04:24,676 | Lambda function 'HelloWorldFunction' is already running
2023-04-12 19:04:24,682 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2023-04-12 19:04:27,695 | Function 'HelloWorldFunction' timed out after 3 seconds
2023-04-12 19:04:28,443 | No response from invoke container for HelloWorldFunction
2023-04-12 19:04:28,443 | Invalid lambda response received: Lambda response must be valid json
2023-04-12 19:04:28 127.0.0.1 - - [12/Apr/2023 19:04:28] "GET /hello HTTP/1.1" 502 -

I have tried a lot of things regarding docker, reinstall AWS CLI and SAM CLI, none worked. I don't know what to do.

Thank you.

tfmeneses commented 1 year ago

I'm getting the same error when I'm running locally using Docker. I can't run the "helloWorld" from sam init/build.

image

Yesterday It was running. Today I installed a Windows Update/Docker Desktop update and it doesn't work more.

image

jfuss commented 1 year ago

@felipeoriani Assuming your issue is the same as @tfmeneses, https://github.com/aws/aws-sam-cli/issues/4222 maybe helpful.

Our dockerpy is pretty out of date, I attempted to upgrade before https://github.com/aws/aws-sam-cli/pull/4356 but ran into some potential back-compat issues and never had the time to dig deeper. It could be upgraded will help this but its hard to know because the error is so value.

@felipeoriani It sounds like what broke is your upgrade. My suggestion would be to downgrade docker desktop and see if that resolves the issue.

tfmeneses commented 1 year ago

@jfuss Thank you very much! It works for me.

felipeoriani commented 1 year ago

@tfmeneses can you share the versions you are using for Windows, AWS CLI, AWS SAM and Docker? I will try it later, thank you!

tfmeneses commented 1 year ago

I'm using:

To solve my problem I disabled the Hyper-V feature.

felipeoriani commented 1 year ago

@jfuss it did not work for me. I downgrade some versions here, and currently using:

I created a new app with sam init using HelloWorld Template for nodejs18.x and Typescript and still got the same error:

2023-04-13 23:45:13 Press CTRL+C to quit
Invoking app.lambdaHandler (nodejs18.x)
Reuse the created warm container for Lambda function 'HelloWorldFunction'
Lambda function 'HelloWorldFunction' is already running
Function 'HelloWorldFunction' timed out after 3 seconds
No response from invoke container for HelloWorldFunction
Invalid lambda response received: Lambda response must be valid json

If you have any other suggestions, I would appreciate it. Thank you.

jfuss commented 1 year ago

@felipeoriani Are you able to run any containers locally? Trying to see if this is only a SAM CLI issue running containers or a wider system thing going on.

felipeoriani commented 1 year ago

Yes, it looks fine on my machine. I am running mysql:8 on my machine. I can make more tests if you want. I have Hyper-V disabled on my Windows.

image

Something I want to try is using WSL2 here. I will prepare a new instance for WSL2 and install AWS CLI and SAM CLI, sharing the docker instance to check if it looks fine.

jfuss commented 1 year ago

@felipeoriani Another thing to try is increase the Lambda timeout. That might be causing some issues if (for some reason) the request to the container is taking too long or something. Just noticed that in your output.

felipeoriani commented 1 year ago

I have tried to set the 30 seconds, but still got the same error. I am thinking it is something on my machine, but I really don't know what. It was working one day, and after I deployed at AWS it was not working on the next day anymore. I know it is weird, but this is the only thing I have done differently. Thanks @jfuss

mndeveci commented 1 year ago

Hi @felipeoriani ,

I've tried to re-produce this issue on a Win11 machine with brand new docker installation using WSL2 (installed newest Docker Desktop which is 4.19) and newest SAM CLI (v1.83). And I got it working without any issues.

While installing Docker Desktop, it warned me to update WSL2 to latest kernel (suggested me to run wsl --update command), you might give it a try for your case.

As @jfuss mentioned and from the command output that you have provided, your invocation times out;

Lambda function 'HelloWorldFunction' is already running
Function 'HelloWorldFunction' timed out after 3 seconds
No response from invoke container for HelloWorldFunction

You might try re-installing the Docker engine, or running docker system prune -a (this command will remove all the images, containers, volumes on your local system, so please don't use it if you have containers or volumes running with critical information) to clean up installation.

Please let us know if any of this works or if you can provide other information for your error.

moelasmar commented 1 year ago

Closing due to inactivity. Feel free to re-open if your issue isn't resolved.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

shivpatel commented 11 months ago

Was having this issue while running sam local on an M1 MacBook (arm64).

By default SAM pulls an x86-based image to run Lambda functions on.

Updating the architecture to arm64 in the template.yml fixed the issue for me.

Architectures:
  - arm64

Hope this helps!

jghaines commented 5 months ago

Was having this issue while running sam local on an M1 MacBook (arm64).

By default SAM pulls an x86-based image to run Lambda functions on.

This should be raised as a separate Issue

ralcorta commented 3 months ago

@felipeoriani Another thing to try is increase the Lambda timeout. That might be causing some issues if (for some reason) the request to the container is taking too long or something. Just noticed that in your output.

That's solve my problem.

Mac

denislapi commented 1 week ago

Could you help me understand what was the fix for this issue?

I tried to use the code from this tutorial and as presented here it should be possible to set the content type.

Is this still an option? If so, do you maybe know why it still expects JSON format even if I specified 'text/plain'

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs20.x
      Architectures:
        - arm64
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

app.mjs

export const lambdaHandler = awslambda.streamifyResponse(
  async (event, responseStream, context) => {
    responseStream.setContentType('text/plain');
    responseStream.write('Hello, world!');
    responseStream.end();
  }
);

Logs

denislapadatovic@Deniss-MacBook-Pro sam-app % sam local start-api
Initializing the lambda functions containers.                                                                                                                                                                          
Local image is up-to-date                                                                                                                                                                                              
Using local image: public.ecr.aws/lambda/nodejs:20-rapid-arm64.                                                                                                                                                        

Mounting /Users/denislapadatovic/Desktop/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                                                                                 
Containers Initialization is done.                                                                                                                                                                                     
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]                                                                                                                                                       
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. If you used sam build
before running local commands, you will need to re-run sam build for the changes to be picked up. You only need to restart SAM CLI if you update your AWS SAM template                                                 
2024-09-10 19:38:04 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
2024-09-10 19:38:04 Press CTRL+C to quit
Invoking app.lambdaHandler (nodejs20.x)                                                                                                                                                                                
Reuse the created warm container for Lambda function 'HelloWorldFunction'                                                                                                                                              
Lambda function 'HelloWorldFunction' is already running                                                                                                                                                                
END RequestId: 7de61b37-fb7d-421f-b4c3-97f233b6184f
REPORT RequestId: 7de61b37-fb7d-421f-b4c3-97f233b6184f  Init Duration: 0.03 ms  Duration: 50.11 ms      Billed Duration: 51 ms  Memory Size: 128 MB     Max Memory Used: 128 MB

Invalid lambda response received: Lambda response must be valid json                                                                                                                                                   
2024-09-10 19:38:15 127.0.0.1 - - [10/Sep/2024 19:38:15] "GET /hello HTTP/1.1" 502 -
2024-09-10 19:38:15 127.0.0.1 - - [10/Sep/2024 19:38:15] "GET /favicon.ico HTTP/1.1" 403 -

Response

image

EDIT:

Versions (MacOS):

EDIT 2:

I tried using the AWS Quick Start application template: 9 - Lambda Response Streaming and got the same result.

EDIT 3:

I tried invoking the Lambda function directly and this is working.

Code:

export const lambdaHandler = awslambda.streamifyResponse(
  async (event, responseStream, context) => {
    responseStream.setContentType('application/json');
    responseStream.write(JSON.stringify({ body: { message: 'Hello, world!' } }));
    responseStream.end();
  }
);

Command:

sam local invoke "HelloWorldFunction" -e events/event.json

Response:

START RequestId: 6fe77a4c-aade-4e70-b4b4-654a930c9af0 Version: $LATEST
END RequestId: 5267f267-5caf-4692-b53e-b9ca59babd99
REPORT RequestId: 5267f267-5caf-4692-b53e-b9ca59babd99  Init Duration: 0.02 ms  Duration: 42.73 ms      Billed Duration: 43 ms  Memory Size: 128 MB     Max Memory Used: 128 MB
{"body": {"message": "Hello, world!"}}

But I still need to figure out how to make it work with API Gateway since when I run the code with sam local start-api

Output:

Lambda function 'HelloWorldFunction' is already running                                                                                                                                                                
START RequestId: 8dfe35dc-e450-40f8-b025-66555079a277 Version: $LATEST
END RequestId: a65f9345-1706-46ee-a3c1-54b1bb8a3193
REPORT RequestId: a65f9345-1706-46ee-a3c1-54b1bb8a3193  Init Duration: 0.04 ms  Duration: 62.78 ms      Billed Duration: 63 ms  Memory Size: 128 MB     Max Memory Used: 128 MB

No Content-Type given. Defaulting to 'application/json'.                                                                                                                                                               
2024-09-11 18:22:55 127.0.0.1 - - [11/Sep/2024 18:22:55] "GET / HTTP/1.1" 200 -
2024-09-11 18:22:55 127.0.0.1 - - [11/Sep/2024 18:22:55] "GET /favicon.ico HTTP/1.1" 403 -

Browser:

image

So I can see the output in the browser, but it is still strange to me why I see No Content-Type given. Defaulting to 'application/json'. even if I set it responseStream.setContentType('application/json');