:bulb: [Feature] Log download stats from THREDDS server

[huard]

Description

It would be useful for reporting purposes to monitor data downloads from THREDDS:

• total volume of downloads (Gb/day);
• per-file download volume (Gb/day);
• per-datasets opendap streaming volumes (Gb/day);

References

This information can be parsed from NGINX logs, but those logs need to be exposed to Prometheus to be aggregated and archived within the current architecture.

Possible solutions:

• https://www.martin-helmich.de/en/blog/monitoring-nginx.html: parses logs and exposes to Prometheus
• Loki (Prometheus for logs, integrates with Graphana), seems possible to configure metrics from logs and expose them to Prometheus
  • https://github.com/nginxinc/nginx-prometheus-exporter: only exposes content from the API, which is very limited
  • https://github.com/google/mtail (general purpose log scraper)

Additional info

• https://www.martin-helmich.de/en/blog/monitoring-nginx.html

See also:

• 12

• 277

Concerned Organizations

[fmigneault]

Consider downloads from WPS outputs and STAC data proxy endpoints as well for the same reasons.

[huard]

ESGF uses Beats and Logstash to collect logs and compute their stats. See https://drive.google.com/drive/folders/1LbvoYeQ_6L_bzTsO-EEhwqjIx1jZ-G1k

[fmigneault]

If the "node collector" can be located on the same instance, logstash seems like an interesting candidate. If there is no distinction between beats or logstash as "log producers", I would favor the 2nd architecture to limit the number of configurations/technologies involved.

[huard]

Parser for nginx logs and prometheus counter https://gist.github.com/huard/25ca5be3479f72546f748da54f7097e7