medic / cht-watchdog

Configuration for deploying a monitoring/alerting stack for CHT
GNU Affero General Public License v3.0
4 stars 7 forks source link

Document process for exposing couch2pg backlog stats without a direct SQL connection #99

Closed mrjones-plip closed 5 months ago

mrjones-plip commented 7 months ago

A best practice CHT Core follows with it's monitoring API is that all metrics are publicly available and do not require authentication to view. This open model breaks a bit when implementing couch2pg backlog monitoring as it requires a password to setup and is intimately tied to your watchdog instance. Instead, it might be loosely coupled to watchdog and more intimately tied to your Postgres server.

We should document the process for how a deployment could easily use the existing Postgres exporter to expose a public endpoint that requires no authentication. The end result is a simple anonymous URL which includes the instance name in the URL/path/query (TBD). We'll use the existing query:

SELECT
  substring(seq for position('-' in seq) - 1) as sequence,
  substring(
    source
    from
      position('/' in source) + 1
  ) as db
FROM
  couchdb_progress
WHERE
  source like '%/%'
  and seq like '%-%'

which returns data like this, but on the postgres exporter URL.

sequence db
47 _users
131899 medic
57960 medic-sentinel
944 medic-users-meta
2279 medic-logs

We can optionally use a reverse proxy which terminates TLS to expose the stats.

Here's the current architecture:

image

This ticket would explore moving the exporter to/closer to the RDBMs

image