search

VertNet / webapp

VertNet web application
8 stars 7 forks source link

Out of Memory exceptions when downloading #624

Open tucotuco opened 8 years ago

tucotuco commented 8 years ago

When downloading,

Trying Query: order:Lagomorpha haslength:1 Options: search.QueryOptions(limit=1000, number_found_accuracy=10000, cursor=search.Cursor(), ids_only=False) Version: search.py 2016-08-15T16:43+02:00

Found 9404 records in 10.1s

App Engine shuts down the instance with a message:

Exceeded soft private memory limit of 128 MB with 138 MB after servicing 26 requests total

This is happening inconsistently, and is something that would be much easier to isolate and purge with a re-architecture to use micro-services or modules.

tucotuco commented 8 years ago

Adding additional gc.collect() statements in the Handlers did not make any difference. Exceptions remain inconsistent, this time with 3 out of ten service/download/write calls causing exception as opposed to 1 before putting in the gc.collect() calls.

Exceeded soft private memory limit of 128 MB with 135 MB after servicing 8 requests total