pmclanahan / django-celery-email

A Django email backend that uses a celery task for sending the email.
BSD 3-Clause "New" or "Revised" License
477 stars 110 forks source link

reduce memory usage when # of emails > chunk size #56

Closed pawl-bb closed 5 years ago

pawl-bb commented 5 years ago

Currently CeleryEmailBackend.send_messages works like this:

  1. All e-mails are serialized with email_to_dict and stored in a list
  2. The list of serialized e-mails is split into chunks
  3. A celery task is created for each chunk

Serializing all the e-mails at once can use lots of memory if you have lots of e-mails with attachments.

To improve memory efficiency, this PR changes CeleryEmailBackend.send_messages to run email_to_dict on one chunk of serialized e-mails at a time.

Benchmarks (on 80 emails with a 5 mb attachment):