This is a pretty bad bug I discovered today. It's the first time I've noticed it because it only occurs if there is a very high volume of incoming emails (it's been steadily going up over the years) and the sidekiq backs up for one reason or another.
What happens is as new emails are created on cuttlefish, the contents of the oldest ones are deleted but it doesn't take into account whether the email was actually sent or not. So, you end up with a bunch of jobs in the background queue that are trying to send emails when the email content has been deleted. This is obviously extremely not great.
So, we need to rethink the EmailDataCache so that it respects whether the email has been sent.
This is a pretty bad bug I discovered today. It's the first time I've noticed it because it only occurs if there is a very high volume of incoming emails (it's been steadily going up over the years) and the sidekiq backs up for one reason or another.
What happens is as new emails are created on cuttlefish, the contents of the oldest ones are deleted but it doesn't take into account whether the email was actually sent or not. So, you end up with a bunch of jobs in the background queue that are trying to send emails when the email content has been deleted. This is obviously extremely not great.
So, we need to rethink the EmailDataCache so that it respects whether the email has been sent.