Add Prometheus instrumentation for ingest-file workers - Githubissues

alephdata / ingest-file

Ingestors extract the contents of mixed unstructured documents into structured (followthemoney) data.

GNU Affero General Public License v3.0

54 stars 25 forks source link

Add Prometheus instrumentation for ingest-file workers #550

Closed tillprochaska closed 9 months ago

tillprochaska commented 9 months ago

This PR adds instrumentation for ingest-file workers:

Upgrades servicelayer to include basic metrics implemented in https://github.com/alephdata/servicelayer/pull/111
Adds metrics related to ingest performance by ingestor (duration of tasks, bytes ingested, number of failed/succeeded tasks)
Adds metric related to PDF cache performance (hits/misses)

In order to test these changes with a local Aleph instance:

Build the ingest-file image using docker build -t ghcr.io/alephdata/ingest-file:metrics ..
In you Aleph docker-compose.dev.yml, set the ingest-file image tag to metrics.
In you Aleph docker-compose.dev.yml, set the PROMETHEUS_ENABLED=true environment variable for the ingest-file service.
Run docker compose -f docker-compose.dev.yml up to apply the changes.
Run docker compose exec ingest-file curl http://localhost:9090/metrics.