investigate monitoring

[pronoiac]

Our instance is on a managed host, so we don't have access to direct monitoring of services. Brainstorming:

• have a heartbeat account
  • likely local
  • Say bong every hour, maybe bing every 5 min, maybe post the time, maybe have an hourly and a five min account
• Poll the rss feed for it
  • maybe last-modified time would work
  • or parse the rss feed

[majick]

Throwing other shit at the wall, in increasing levels of complexity.

• Try to get a 2xx or 3xx, get mad and emit an angry noise if you get a 4xx, 5xx, or curl exiting with ret 6? -- relevant to 20230114 but maybe not the prior. I dunno, I'm new.
• Instrumentation? Is instrumentation a thing? Is this codebase instrumented in some way? Like, with metrics?
• Instrumentation? Is instrumentation a thing? Is the underlying platform instrumented in some way? Like, with metrics?
• How heinous is the API? Is it stable enough or non-annoying enough to have some kind of basic SLI against?

Is there a box from whence to run these things? Is one needed?

[pronoiac]

Dear Reader, before you go further, you should probably know that I'm an SRE - Site Reliability Engineer - and so this falls deeply into an area of interest for me, and so I can go on in waaay too much detail on this sort of thing. sorry not sorry.

For the 2022-12-28 outage, the site was still responding, but updates were (mostly) stalled. (In fact, I'm not absolutely positive the rss feeds weren't updating, but it seems like a safe bet.) For the 2023-01-15 outage, the site responded with 504s. (Oops, I haven't pushed the "we're back up" edit.)

I'm not sure yet what it looks like if a request hits during an upgrade. They're expected to take about a minute, so successive requests - even 5 min later - would be expected to be ok. We'd alert on repeated errors.

I should look into more official metrics, and what hosting offers. I jumped into "treating this as a black box" (possibly too) readily, as I built something similar at a previous job. ... from that experience, I'd suggest, as separate metrics:

• success or failure: 2xx/3xx vs 4xx/5xx
• latency of request, measured in ms
• latency re: posted timestamps, measured in min

I already have a personal VPS running Docker, to start with, though sharing access is currently non-trivial.

Because I had to refresh my memory on this:

• SLO, Service-Level Objective - our target goal
• SLA, ... Agreement - below this level of e.g. uptime, penalties are paid. Not applicable here.
• SLI, ... Indicator - measurements of metrics

[majick]

My friend, let us enjoy sitting together companionably in this wheelhouse! I am the SRE they send to make other people's RCAs meatier/less-blamey/more-actual-rootley, so, also, yes, very sorry-not-sorry to enjoy being in this place at this time.

The way I've been pushing SLIs is as "measure of what a person I care about actually gives a crap about" and SLOs as "how hard they give a crap about it." SLAs are, for me, eh, more about setting release policy or revenue clawbacks or clubbing executive people over the head for budget and other non-applicable stuff.

From that, you can derive all kinds of cool stuff without hanging to stand up a TSDB pod and Grafana and all that -- which, given my personal proclivities, would probably be what I'd actually wind up doing myself. Instead you can just say:

• User cares about site up. SLI: Please emit 2xx/3xx. SLO: Do that pretty much every time you curl it from some cron job.
• User cares about raw arbitrary endpoint "fast enough." SLI: Dumbest fastest health check request. SLO: Milliseconds not seconds.
• User cares about Site Capabilities X, Y, Z - I dunno, start with the broad swath? Reads of most common object (toot?) and writes of same?

This is looooots of fun for me to deep dive on, especially in ways that are actively useful as opposed to potentially bikesheddy, so please do feel free to ask that be amped or attenuated.