Miserlou / Zappa

Serverless Python
https://blog.zappa.io/
MIT License
11.89k stars 1.21k forks source link

Random 'NoneType' object is not callable #795

Open bxm156 opened 7 years ago

bxm156 commented 7 years ago

Randomly my site will go down (perhaps once or twice a month) and display the following: { "message": "An uncaught exception happened while servicing this request. You can investigate this with the zappa tail command." }

The logs say the following:

[1492620032065] 'NoneType' object is not callable
[1492620147882] 'NoneType' object is not callable
[1492620330874] 'NoneType' object is not callable

etc. I don't see any further information in the logs that would be helpful. Is there a way I can get more detailed information about the error? It only happens about once a month or less. When it get into this state, the only thing I found has fixed the problem is redeploying the code.

Other side notes: My Django project uses Amazon RDS. I can see in the graph that during this time, the DB connections goes to 0. And then back up once I have redeployed the code.

## Possible Fix

Re-deploying my lambda code fixes the issue.

Steps to Reproduce

I haven't been able to reproduce it consistently, it just will happen randomly and won't be fixed until I redeploy my code.

Your Environment

pawel-furmaniak commented 4 years ago

This error can be caused by this SQLite problem: https://github.com/Miserlou/Zappa/issues/1880

Please check zappa logs during deployment with zappa tail <ENV_NAME>

nabazm commented 4 years ago

not using sqlite at all, not sure if Django keep some reference for it somewhere

jneves commented 4 years ago

@nabazm I think your case is just a repeat of #1638

jneves commented 4 years ago

I'm keeping this one open, because the damned message is not clear (and I hope to sort it out). I think I've fixed the problem with the "slim_handler: true" error handling that was causing corrupt files last year. Most other cases tend to fall into:

  1. temporary initialisation failures - Sometimes is just a temporary network connection issue, increasing the timeout will help with those corner cases. In my experience, sometimes you'll get a container from AWS Lambda with networking issues or slowish, so also take into account that can happen, but it's rarer these days.
  2. permanent initialisation failures - When you get this message consistently. If the lambda is running in a VPC, use VPC endpoints for S3 buckets (remote_env and others) - https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/. Also check your permissions and dependencies. If you're deploying from an environment that is not binary compatible with AMI Linux, check if packaging with docker lambci images helps. Another source of timeouts is policies that don't cover all the permissions needed (s3 buckets, database connections or other AWS services, http requests to other services without having a NAT gateway).

That's all I can think of at the moment. If you can replicate something outside or have an idea to make the information clearer, please share.

dleber commented 4 years ago

In my case, the issue was caused by:

Solution: Increase memory_size and/or timeout_seconds.

These are the steps I took:

Thanks to @eexwhyzee for his comment prompting me in the right direction. Also in general, it's a good idea to check metrics/logs in the lambda console or cloudwatch. This revealed that memory was maxing out.

ianseyer commented 4 years ago

Also getting this error.

Deploying a flask application that uses boto3.

Have a single POST route, slim_handler is false and there is sufficient memory (512mb).

Very frustrating.

nabazm commented 4 years ago

@nabazm I think your case is just a repeat of #1638

In my case was needed to add vpc endpoints for s3 and the issue resolved,

sanhardik commented 3 years ago

I am seeing issue without deploying. I am thinking its failing when its deploying the code to new containers.

I don't have slim_handler in my zappa settings, so I am assuming that is set to False (Default)

I have a keep_warm callback at 4 mins interval.

I am not able to see the error in the Cloudwatch Logs. Any ideas ?

yifaneye commented 3 years ago

I also got this error. In my case, I found I had a wrong import statement of a module after running zappa tail. After fixing the import statement, this error was resolved.

Ati59 commented 3 years ago

Also getting this error deploying a flask application. slim_handler to true too

justinwride commented 2 years ago

I recently experienced this error today when updating some dependencies in requirements.txt for a Django application.

zappa update was working previously, but started giving me the NoneType object is not callable error after changing some versions of packages.

I wasn't able to find the exact package that was causing the issue, but I believe it was psycopg2-binary. I changed it back to psycopg2-binary==2.8.6 and the error was fixed.

I've heard that using aws-psycopg2 can help.

iperera97 commented 1 year ago

solution for me dont use sqlite as the db use another db like mysql or postgresql