Request Timeouts and hitting memory quota on Heroku (Memory leak?)

[J-Gonzalez]

After running the app for a few hours on Heroku, we start seeing a ton of request timeouts, and eventually start using up a ton of memory on Heroku.

Aug 04 10:16:05 my_image_server heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/w352/eventFlyer/1437145123270-facb962019235ea60d67df94.jpg" host=my_image_server.herokuapp.com request_id=dd4ee711-bd31-4dbf-91c2-5b977bac94c7 fwd="SOME IP ADDRESS" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0 
Aug 04 10:16:05 my_image_server heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/w352/eventFlyer/1437434594799-b4b4d128fb48871cc8a4ce87.jpg" host=my_image_server.herokuapp.com request_id=c67ed129-41e0-4afb-b1ed-92227e47e56f fwd="SOME IP ADDRESS" dyno=web.1 connect=2ms service=30000ms status=503 bytes=0 
Aug 04 10:16:07 my_image_server heroku/web.1:  Error R14 (Memory quota exceeded) 
Aug 04 10:16:07 my_image_server heroku/web.1:  Process running mem=2521M(492.4%) 
Aug 04 10:16:08 my_image_server heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/w900/eventFlyer/1437794831120-a15860af865d08c4d83b135d.jpg" host=my_image_server.herokuapp.com request_id=f360964f-dc4c-4c4b-ac03-86a6b69d03a3 fwd="SOME IP ADDRESS" dyno=web.1 connect=2ms service=30009ms status=503 bytes=0 
Aug 04 10:16:08 my_image_server heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/s80/eventFlyer/1438707312555-a8b1f5fee9d503055e010ae4.jpg" host=my_image_server.herokuapp.com request_id=87d14225-d945-4de9-9bf2-28792077e945 fwd="SOME IP ADDRESS" dyno=web.1 connect=13ms service=30005ms status=503 bytes=0  
Aug 04 10:16:08 my_image_server heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/w352/eventFlyer/1437794831120-a15860af865d08c4d83b135d.jpg" host=my_image_server.herokuapp.com request_id=e27688fe-0fa8-43a1-81b4-44593ee9742b fwd="SOME IP ADDRESS" dyno=web.1 connect=0ms service=30002ms status=503 bytes=0 

[J-Gonzalez]

The 'Memory Quota Exceeded' could be due to the fact that we are buffering the entire objects into memory, and with a ton of images being generated, this would add up quickly - not sure if something like this would help: https://github.com/aws/aws-sdk-js/issues/656

[jimmynicol]

@J-Gonzalez,

A few thoughts here:

• are you running behind a CDN? much of the premise of this app is lost when no CDN is used.
• from the logs it looks like you are using a 1X dyno which may not be sufficient if your source images are very large.
• so very early on the code was attempting to stream objects as opposed to buffering them to memory but it turned out that gm was simply buffering them anyway and sharp may do something similar as some of the operations require all of the image to be used.

It is possible there could be a memory leak in the code, but my first guess is that you are asking the simple dyno to do too much work. Try scaling to a 2X and see if things improve, but as I said it is fairly dependant on your source images.

[J-Gonzalez]

Scaling it to 2X did seem stop memory leaks. The app does climb to about 700MB of memory usage but doesn't seem to go above that.

[lovell]

Hello, you may want to reduce the size of libvips' operation cache on memory-constrained systems - see http://sharp.dimens.io/en/stable/api/#sharpcachememory-items

[jimmynicol]

@lovell, thanks again for the input.

@J-Gonzalez if you can give this a go I would love to hear the results. image-resizer is designed to have the most sensible defaults for most cases and it sounds like setting appropriate cache sizes should be one of them.

[jimmynicol]

Closing due to inactivity.