search

Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
694 stars 65 forks source link

sharding: made metrics buffer bigger #268

Closed shuhaowu closed 3 years ago

shuhaowu commented 3 years ago

I'm seeing metrics sink full warnings in production spamming our logs. I tested making the buffer bigger and it appears to have helped...

That said, I'm not sure why making metrics buffer bigger actually solves the problem, because there's should be a "conservation of mass" type dynamics: the number of metrics going into the channel should equal the number coming out for the channel to not be filled. I guess buffers for audio workstations also have similar behaviours, so maybe I should just read up more about how buffers work.

shuhaowu commented 3 years ago

This is just the datadog/statsd metrics that we get. When this channel gets full, a warning is emitted by Ghostferry. If it gets full fairly often, the ghostferry logs in stderr gets spammed. I'm not sure what you mean by payload?

I'm not even sure the metrics are being used anywhere, but i didn't want to break it for now if I can resolve the log spam issue. Also statsd metrics are best-effort anyway, as my understanding is that it is transmitted via UDP.