Persist throws errors and drops messages when writing to AWS

inhindsight / hindsight

Apache License 2.0

12 stars 8 forks source link

Persist throws errors and drops messages when writing to AWS #196

Open ManApart opened 4 years ago

ManApart commented 4 years ago

Given 10 million small messages to persist, Persist constantly threw SSL errors and dropped 115,064 messages.

We lost the log of this issue. Will try to grab it and update next time we recreate the issue

[ ] Persist should not throw errors when writing to AWS
[ ] Persist should not drop messages

jessie-morris commented 4 years ago

My guess is this is seen because of the differences in communicating with minio and /real/ aws s3. I might split this card into the two different problems as I do not think these are for sure related, but that might be more valuable to do when that is known.

ManApart commented 4 years ago

I'm all for splitting this once we get ahold of the log again so we have some more detail.