Out of memory errors on App Engine

[GoogleCodeExporter]

We use feedparser 5.1 in our app running on Google App Engine, and we parse 
hundreds of thousands of feeds on a regular basis. But I've come across a few 
feeds that have proved problematic. For example, this feed (copy attached):

http://www.moneysupermarket.com/shopping/blog/feed/

    The feed size is about 0.5MB. It's big but not huge. When I run the parser on my local machine with plenty of RAM it parses without problems. But when I run it on App Engine instances with 128MB of RAM (i.e. the amount of RAM available to the app), the instance runs out of memory and gets killed by the system. I tried bigger instances with 256MB of RAM and those failed as well. In comparison, when parsing other feeds, the instance memory usage rarely goes above 60MB, but with this feed it jumps to 300MB or more before getting killed. 

    I don't think feedparser typically needs as much RAM as 50X the feed size, so I'm guessing there might be a problem somewhere.

Original issue reported on code.google.com by wal...@ninua.com on 16 Dec 2011 at 9:19

Attachments:

• out_of_memory_feed.xml

[GoogleCodeExporter]

I haven't forgotten about this! I've been trying to figure out how to measure 
the memory usage while the application is running. Some memory profilers 
haven't been updated in years, and others seem to have sparse or no 
documentation, and still others won't compile on my machine, but I'm working on 
it!

Original comment by kurtmckee on 27 Feb 2012 at 4:05

• Changed title: Out of memory errors on App Engine
• Changed state: Accepted

[GoogleCodeExporter]

Thanks, Kurt. I tried with many tools as well and got into the same issues. 

Original comment by wal...@ninua.com on 27 Feb 2012 at 6:42