search

Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
693 stars 65 forks source link

Consume and send metrics to statsd in copydb #311

Closed tiwilliam closed 2 years ago

tiwilliam commented 2 years ago

This PR will configure copydb to consume metric events and dispatch them to statsd (datadog-go/statsd flavoured). The consumer in sharding have been cleaned up and tested.

I had to upgrade testify to get access to assert.PanicsWithError.

Breaking change (sharding only)

It's now also possible to disable consumption of metrics if StatsdAddress isn't configured, this is a breaking change if you rely on the default fallback to 127.0.0.1:8125. If you are using this fallback, configure StatsdAddress to 127.0.0.1:8125 in your config.

The metric prefix have also been updated to include ghostferry.

Old metric key: sharding.RowEvent New metric key: ghostferry.sharding.RowEvent

Suggested new metrics

ghostferry.copydb.BinlogStreamer.GetEvent (timing, tags=[DeadlineExceeded:bool])
ghostferry.copydb.BinlogStreamer.RotateEvent (incr, tags=[FakeRotateEvent:bool])
ghostferry.copydb.BinlogStreamer.FormatDescriptionEvent (incr, tags=[EventPositionValid:bool])
ghostferry.copydb.BinlogStreamer.RowsEvent (timing, tags=[])
ghostferry.copydb.BinlogWriter.WaitOnBuffer (timing, tags=[])
ghostferry.copydb.BinlogWriter.WriteEvents (timing, tags=[])
ghostferry.copydb.BinlogWriter.UpdateLastResumableSourceBinlogPosition (timing, tags=[])