CHT App Monitoring Data Ingestion

Purpose

Scrapes data from CHT instances and ingests in Postgres. The resulting data powers our App Monitoring dashboards and can answer questions like:

• How many projects are running cht-core version 2.x or 3.x? How many are on the latest version?
• Which projects are using declarative config?
• Which projects use outbound push? How many projects enable the death_reporting transition?
• How many users have the can_logout permission?
• How many production deployments happened this month? How many forms changed in production this month?
• How have data from the monitoring API changed over time?
• Which projects upgraded this month?

Configuration

Database

You will need an instance of postgres available and a username and password with access to a database that has these tables:

CREATE TABLE public.monitoring_urls (
    id SERIAL PRIMARY KEY,
    url VARCHAR NOT NULL,
    partner_name text,
    enabled BOOLEAN NOT NULL DEFAULT true,
    access_level smallint NOT NULL DEFAULT 1
);

CREATE TABLE public.monitoring_docs (
    id SERIAL PRIMARY KEY,
    url_id INTEGER NOT NULL,
    doctype VARCHAR NOT NULL,
    created TIMESTAMP DEFAULT NOW(),
    doc JSONB NOT NULL
);

Medic App Services teammates should use the postgres/db_schema.sql instead of the above schema.

After you have created the tables above, the list of CHT instances to scrape is controlled via the Postgres table monitoring_urls:

• To add a new instance, insert a row into the table
• To disable an instance, set enabled to false
• To scrape deep metrics for an instance, set access-level to 1, 2, or 3

Credentials

Credentials to write to the monitoring_docs table and read from the monitoring_urls us set with these 5 environment variables. The MONITORING_USERNAME and MONITORING_PASSWORD variables are shared and used to log in to all CHT instances. You therefore need to set the same username and password for all CHT instances you want to monitor using authenticated logins.

If you are only using access_level of 1, you do not need to set MONITORING_USERNAME and MONITORING_PASSWORD as the CHT monitoring API does not require any authentication.

Copy the .env.example file to .env and edit it with your deployment's values. It will look something like this:

PG_PORT=5432
PG_USERNAME=postgres
PG_PASSWORD=abc123
PG_HOST=localhost
PG_DATABASE=impact
MONITORING_USERNAME=app-monitor
MONITORING_PASSWORD=app-monitor

You can then use this file for both docker and node based deployments.

Accessing remote Postgres instances

If you need to access a Postgres instance via an SSH tunnel, consider using autossh. This can be installed with apt or snap on Ubuntu and will ensure the tunnel is up at all times to ensure consistent access to Postgres.

If you need to access a remote Postgres instance from inside a docker container over an SSH tunnel, consider binding the tunnel to the docker subnet, which is normally a /16 and is often 172.17.0.1/16. This might look like:

ssh -L 172.17.0.1:5432:localhost:5432 username@remote.db-server.org 

You would then set PG_HOST to 172.17.0.1 and it will be accessible to all containers. See this blog post for more info. autossh can be used in combination with this technique.

Execution

Both the node and docker versions need to be run on a schedule, as natively the app will run once and quit. Consider using cron.

Node

1. Ensure you have node and npm installed ( nvm is an easy way).
2. Confirm node and npm are installed with node -v&&npm -v
3. Clone this repo and cd into the app-monitoring-data-ingest directory
4. Install dependencies with npm ci
5. Set your environment vars with source .env
6. Running the app with node .

Docker

For a fully docker based deployment, you can set up CHT couch2pg first, and then add a database and schema from above to the Postgres instance.

After you have set up your .env per above, run docker-compose up. All related should be built and downloaded as needed.

Output

As the access-level of the user increases from 1 up to 3, richer metrics are available. The results of the scraped data are stored in Postgres tables:

Access Level 1

Access Level 2

1. Login as administrator
2. Create new doc in _users database

{
  "_id": "org.couchdb.user:app-monitoring",
  "name": "app-monitoring",
  "type": "user",
  "roles": ["app_monitoring"],
  "password": "abc123"
}

3. Create new doc in medic database

{
  "_id": "org.couchdb.user:app-monitoring",
  "name": "app-monitoring",
  "type": "user-settings",
  "roles": ["app_monitoring"]
}

Access Level 3

Access Level 2 but add role mm-online. This gives the user access to all data on the instance.

Analysis

This is the structure of documents in monitoring_docs with doctype='analysis':

Dependencies

Monitors projects using cht-couch2pg v3.6.2 and above.