Monitoring epic, v3 - Githubissues

ooni / backend

Everything related to OONI backend infrastructure: ooni/api, ooni/pipeline, ooni/sysadmin, collector, bouncers and test-helpers

BSD 3-Clause "New" or "Revised" License

51 stars 29 forks source link

Monitoring epic, v3 #243

Open darkk opened 6 years ago

darkk commented 6 years ago

[ ] monitor Oops and BUG: in kernel log ooni/sysadmin#210
[ ] add regexp for go app crashes in systemd syslog ooni/sysadmin#220
[ ] monitor nginx error.log ooni/sysadmin#138
[ ] monitor public endpoints with non-trivial health check ooni/sysadmin#113 ooni/sysadmin#173 ooni/sysadmin#194 https://github.com/ooni/sysadmin/pull/277#issuecomment-469057510
[ ] monitor public webservice health check from another prometheus ooni/sysadmin#174
[ ] monitor all network endpoints with blackbox checks -- 373600887c24eab683d3e918fb45b33cd1d720a6
[ ] postgres replica latency ooni/sysadmin#240
[ ] anonymize IPv4 address to /24 and IPv6 to /48 (min routable prefix: 1, 2 while sou, ) -- 7b12322deb8d217e3ad37d6faea14f8d796a20f8, ea1f6a4826f537b9b725c39800c522a666714cc4
[ ] use dockerd monitoring to track fast restarts with engine_daemon_container_actions_seconds_count{action="start"} ooni/sysadmin#220
[ ] monitor number of systemd service restarts-per-minute (needs node_exporter >0.16.0)

darkk commented 5 years ago

BTW, it's relatively easy to monitor tor hidden service within current setup. The plan is:

re-deploy amstorth.ooni.nu from scratch to clean it up (it was ancient ooni measurements dev/test box)
add DNSPort and TransPort to tor there
put blackbox_exporter there under separate uid
redirect blackbox_exporter traffic to tor (either all, or onion-only)

darkk commented 5 years ago

Actually, I'm very unsure if all of the above makes sense right now. I've observed that capacity and rare-errors alerts at #ooni-bots are often neither acted upon not silenced, so they turn the channel into stream of noise. I assume that adding more of that would be counterproductive with current alerting setup.

So, given recent incidents, I'm limiting it to monitoring of public *.onion endpoints of OONI at this moment as it's both a non-trivial and needed check.

hellais commented 5 years ago

I agree that it's more important to conclude the other pipeline related tasks and the current state of monitoring doesn't warrant for more noise.

Speaking of which @darkk is it possible to reduce the level of noisiness of the onion service alerts? I have seen them get pretty chatty recently.

darkk commented 5 years ago

Well, they were added recently and were consistently failing :-) Let's try to bump timeouts: 36f0d85aeaa5b45b5d3db146be21839fb2ba5edd, but maybe that's some issue with publication of onion service descriptors or something else that may be worth investigation.

hellais commented 5 years ago

I see this has been moved into the "Ready for Review" column. I guess there is still some stuff that we may want to do related to this and it's probably wise to keep the issue open.

Perhaps the best course of action is to unassign it from you and remove it from the Leonid plan board?

darkk commented 5 years ago

Yes, that was exactly the point.

Implementation of the items mentioned in the ticket may be valuable in the future. The current value of those items is unclear (I'm unsure who will react to those alerts and what the reaction should be) and that's why they were postponed at the first place. I think, the important thing to do before going further with moar-alerts is to build a better process around alerts handling.

So, as I've said, I've limited my activity to monitoring *.onion endpoints as 1) it is not so trivial to set-up, 2) it is important as those public endpoints going down lead to data loss according to https://github.com/ooni/sysadmin/pull/277#issuecomment-469057510

hellais commented 4 years ago

cc @FedericoCeratto