flipp-oss / deimos

Framework to work with Kafka, Avro and ActiveRecord
Other
60 stars 22 forks source link

Option to skip large messages on DB poller #210

Closed dorner closed 8 months ago

dorner commented 8 months ago

Currently, when DB Poller is unable to publish messages because they're too large for the broker, the poller will retry forever. This might be "correct" if we need some kind of alert / manual fix for the data. But in most cases, we'll want to just log an error and metric and move on.

This PR adds a configuration so we can make that happen.