domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
961 stars 209 forks source link

Replace multiprocessing.Pool with Pipe + Process #489

Closed Szasza closed 3 months ago

Szasza commented 3 months ago

The multiprocessing library's Pool is used for the reports' parallel processing. This by default shouldn't cause an issue, but unfortunately there are some runtime environments, such as AWS Lambda, where /dev/shm (shared memory for processes) doesn't exist.

To address the above, it would be beneficial to change the multiprocessing.Pool implementation to use Pipe + Process instead, so that the multi-core/multi-process handling is more widely supported.

I am happy to work on an implementation.