search

livepeer / catalyst-api

MIT License
4 stars 2 forks source link

Improve handling Kafka write errors #1344

Closed leszko closed 1 month ago

leszko commented 1 month ago

Improvements:

  1. Increase the internal buffer size from 5000 analytics logs to 25000 analytics logs (this will delay the 5xx to more Kafka write errors before we start to see them)
  2. Update warning message to not include a word error
  3. Add retries to writing messages to Kafka

Fix https://linear.app/livepeer/issue/PS-721/5xx-alerts-sto-prod-catalyst-4lp-playbackstudio-prod-catalyst-4

linear[bot] commented 1 month ago

PS-721 5XX Alerts (sto-prod-catalyst-4.lp-playback.studio prod-catalyst-4)

pwilczynskiclearcode commented 1 month ago

Fine as a quickfix but would be good to have LogChannelBufferSize and kafkaWriteRetries as configuration options

leszko commented 1 month ago

Fine as a quickfix but would be good to have LogChannelBufferSize and kafkaWriteRetries as configuration options

Do you mean like a flag of catalyst-api?

pwilczynskiclearcode commented 1 month ago

Fine as a quickfix but would be good to have LogChannelBufferSize and kafkaWriteRetries as configuration options

Do you mean like a flag of catalyst-api?

Yes.