search

leafac / kill-the-newsletter

Convert email newsletters into Atom feeds
https://kill-the-newsletter.com
MIT License
2.31k stars 113 forks source link

Race condition when receiving multiple emails for the same feed #6

Closed atduarte closed 5 years ago

atduarte commented 5 years ago

Symptoms: Created a kill-the-newsletter.com feed that every 24h receives 5 new emails. Multiple successfully sent emails never show up in the feed.

Possible explanation: Every time a email arrives exim creates a new Rails runner. (https://github.com/leafac/kill-the-newsletter/blob/master/config/exim/exim.production.example.conf#L29)

This runner reads a file, modifies it and then rewrites it. (https://github.com/leafac/kill-the-newsletter/blob/97cbc83c589033197370ceecb850c587d0399d4a/app/mailers/inbox_mailer.rb#L29)

If two or more emails arrive (more or less) at the same time we have a race condition that may results in not persisting all but one email.

leafac commented 5 years ago

Unfortunately Kill the Newsletter! dropped some messages between yesterday and today, which may explain why you don’t see all the emails in your feeds. But the problem should be fixed now.

In any case, you’re right that a race condition may occur, but it should be pretty rare because delivery is very fast. Please let me know if you continue to see problems, in which case I may install some sort of lock or serialize the delivery with Exim’s configuration max_parallel = 1.

leafac commented 5 years ago

After running Kill the Newsletter! for many years, I finally observed one case of a race condition. I fixed the affected feed manually and fixed the race condition for good.