Container Image runs fine with AWS Lambda Runtime Interface Emulator, but 502 on zappa deploy

padmanabhankrishnamurthy commented 2 years ago

Context

I followed the steps here to create a container image, upload it to ECR, and run zappa deploy -d. All routes work perfectly when I test out the image with the AWS Runtime Interface Emulator, i.e, curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"path": "/", "httpMethod": "GET", "requestContext": {}, "body": null}', but zappa deploy -d throws me a 502 error. I've run zappa tail, but all I see there is [1659730496257] 'NoneType' object is not callable.

Unfortunately I'm not cleared by my company to release more than the settings.json file here, but happy to share more details privately.

I have a few questions:

How is it possible that the image works when being tested on the runtime interface emulator, but fails during deployment? I've rewritten the settings file and made sure that slim_handler is set to false as well.
Are there any suggestions on where to embed print statements in the handler.py to make zappa tail more informative during debugging?
Given that I'm deploying this as a docker image, I am not using a venv, and invoking zappa from my project's conda environment. Could this be the root of any issues?

Expected Behavior

zappa deploy -d executes successfully and returns the API endpoint.

Actual Behavior

NoneType object is not callable. Same message replicated in zappa tail. But the same image works when tested with the AWS local Runtime Interface Emulator

Your Environment

Zappa version used: 0.55.0
Operating System and Python version: macOS 12.4 Monterrey, Python 3.9
The output of pip freeze: Running everything in docker, so shouldn't be an issue
Link to your project (optional):
Your zappa_settings.json: { "lambda_docker_flask": { "app_function": "server.app", "project_name": "app", "lambda_description": "Zappa + Docker + Flask" } }

monkut commented 2 years ago

502 often results from a problem in the code. Make sure that the code can be run before deployment

padmanabhankrishnamurthy commented 2 years ago

It definitely can - the code is running locally via a standard python3 script.py call, and even using AWS' runtime interface emulator via a curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"path": "/", "httpMethod": "GET", "requestContext": {}, "body": null}'.

Zappa deployment is the only time it looks to be failing.

monkut commented 2 years ago

Just to be sure I would test the code outside of the lambda environment. It's possible that the lambda integration isn't hitting the code as expected.

padmanabhankrishnamurthy commented 2 years ago

Thanks! Tried that too; ran the code outside the docker container, in a different venv, and also on a different machine entirely - no bugs there.

padmanabhankrishnamurthy commented 2 years ago

Update for anyone facing similar issues: FIXED

I try/except'ed every import statement in my app and printed out all failures. Turns out all my issues were caused because of module import errors. After resolving those import paths, my app loads successfully. Not sure why this had to be done since Dockerizing the app shouldn't have allowed import errors to exist (or am I mistaken?).

I'm now facing 504 timeout errors, but that's a story for another day.

misterjunio commented 2 years ago

@padmanabhankrishnamurthy did you figure out the issue you had with the 504? From the CloudWatch logs I can tell my Lambda is starting up correctly, but for some reason it runs all the way to timeout at 30 seconds instead of returning my "hello world"

EDIT: Nevermind, I was playing around with multiple Docker images and realised I was deploying the wrong one 🤦

padmanabhankrishnamurthy commented 2 years ago

Good to know!

For anyone browsing this thread looking for ideas about the 504 error - in my case, my app.py script was importing a ton of heavy libraries. Just importing the libraries and instantiating a few global objects was taking ~60s, by which time a timeout error was thrown. Moving the imports and instantiations to the route instead of keeping them global, and also using the task async decorator solved the issue for me.

zappa / Zappa