search

haystack / YouPS

YouPS: An email automation tool that makes your email management easy! 😎
11 stars 5 forks source link

Memory Issues #116

Closed lukesmurray closed 5 years ago

lukesmurray commented 5 years ago

I've traced down the memory issues on the initial sync to a single method

https://github.com/soyapark/murmur/blob/cd8f44ab3f6e999a7c6839c3919c69a230366e0d/engine/models/folder.py#L259-L377

There is a known memory leak in django when django is in debug mode. Django stores every single query in connections.queries variable. In order to fix that issue I've started using bulk_create to create objects and django.db.reset_queries to remove cached queries.

The second option I've been exploring is whether or not Django caches too much information. To get around caching I've been using bulk_create and MessageSchema.objects.iterator

soyapark commented 5 years ago

@lukesmurray I think this part is downloading entire messages including body and attachments due to the descriptor Envelope. https://github.com/soyapark/murmur/blob/330389e1674d611723491b579b5ca17722a4074c/engine/models/folder.py#L280-L283

lukesmurray commented 5 years ago

These have been mostly fixed. Reducing memory further could be done by batching our application as much as possible.