flipp-oss / deimos

Framework to work with Kafka, Avro and ActiveRecord
Other
59 stars 22 forks source link

Update send_produce_error to decode failed_messages with built-in decoder. #76

Closed DeeChau closed 3 years ago

DeeChau commented 4 years ago

Description

Fixes #19 - Update send_produce_error to decode failed_messages with built-in decoder.

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Tested on a staging environment of fadmin. Metrics show up in datadog tracking when errors happen during the delivery of messages to kafka. i.e

Failed to send all messages to multi.Flyers.FlyerDataResponse/2; attempting retry 1 of 10 after 5s
Failed to fetch metadata from kafka://kafka-stg.bootstrap.wishabi.net:9093: Connection error Errno::ETIMEDOUT: Connection timed out

Checklist:

DeeChau commented 3 years ago

Addressed comments and tested on a staging environment. Metrics show up in datadog tracking when errors happen during the delivery of messages to kafka. e.g.

Failed to send all messages to multi.Flyers.FlyerDataResponse/2; attempting retry 1 of 10 after 5s
Failed to fetch metadata from kafka://kafka-stg.bootstrap.wishabi.net:9093: Connection error Errno::ETIMEDOUT: Connection timed out

have corresponding metrics in received. image

CloudWatch Logs Insights
region: us-east-1
log-group-names: fadmin-multi
start-time: 2020-08-04T18:31:31.000Z
end-time: 2020-08-04T20:50:23.000Z
query-string:

filter @message LIKE /Failed to send all messages to/ 
| filter @message LIKE /keeping remaining messages in buffer/
| sort @timestamp desc
| parse '[7150]: Failed to send all messages to */' as topic
| stats count() by topic

topic count()
multi.Flyers.FlyerFsas 12
multi.Flyers.FlyerDataResponse 10
multi.fadmin_flyers 8

DeeChau commented 3 years ago

@dorner Made the changes!

dorner commented 3 years ago

Nice, thanks!