search

flightstats / hub

fault tolerant, highly available service for data storage and distribution
http://www.flightstats.com
MIT License
103 stars 35 forks source link

add more statsd event logging around hub startup #1214

Closed lkemmerer closed 5 years ago

lkemmerer commented 5 years ago

We have a statsd event in the shutdown manager (https://app.datadoghq.com/event/stream?tags_execution=and&show_private=true&per_page=30&query=tags%3Acluster%3Aiad.staging%2Crestart%20hub%20priority%3Aall%20status%3Aall&aggregate_up=true&use_date_happened=false&display_timeline=true&from_ts=1561553760000&priority=normal&live=true&is_zoomed=false&status=all&to_ts=1561568160000&is_auto=false&incident=true&only_discussed=false&no_user=false&page=0&bucket_size=240000) , but we don't have anything that tracks startup.

I was debating between "events" and using a counter of some sort so we can create an alert on high rates of restarts. This way follows the existing pattern... What do you think?

Paul-Hess commented 5 years ago

event is datadog specific so there may be value in moving away from that pattern and using a counter

lkemmerer commented 5 years ago

Cool, I think that'll be more useful. I'll close this and do a new PR that switches over to a pattern we can use with TICK and, more generally, add monitoring/alerting to.