search

ResEnv / chain-api

RESTful sensor API
MIT License
39 stars 16 forks source link

server should monitor and report events #29

Open ssfrr opened 9 years ago

ssfrr commented 9 years ago

we have a graphite instance running to track basic system stats (hard drive space CPU usage, etc.), and a statsd server so it's easy to pump new data in. I'd like to start keeping track of the rate new data is coming in, error rates, etc.

kkleidal commented 9 years ago

This seems like a good task for a "topics" network flow: whenever a request is made/data is posted, we could call a method which writes the event details to a websockets (or ZMQ) stream with a tag. Then, we could have an analyzer microservice monitoring the stream and logging as necessary.

ssfrr commented 9 years ago

I think it's even easier than that. We already have the graphite instance up and receiving data, so all we need to do us use a statsd client from python to pump metrics in.

Statsd works with a tagging system sort of like what you're describing so you can label your metrics, but we can just instrument the code so that we send the event signal when the things we're tracking happen. It sends over UDP so it doesn't really add overhead even if the receiving server goes down.