[Epic] Metrics for mediawiki/wikibase

[dajuno]

wiki metrics such as number of items/pages etc should be tracked for prometheus/grafana, or, alternatively, statsd+graphite.

TODO:

• determine relevant metrics (see https://grafana.wikimedia.org/?orgId=1 for inspiration):
  • [ ] relevant metrics for math research data
  • [ ] relevant portal usage metrics

Metrics scraper and visualization:

• [x] prometheus installed (metrics provider) -> https://prometheus.portal.mardi4nfdi.de
• [x] graphana installed (data visualization) -> https://graphana.portal.mardi4nfdi.de

List of services of interest:

• system monitoring:

  • [x] traefik -> frontend traffic (prometheus provider + graphana dashboard)
  • [x] node exporter -> system/hardware metrics (prometheus provider + graphana dashboard, storage space alerts)
  • [ ] #213
  • [x] watchtower metrics https://containrrr.dev/watchtower/metrics/
  • [x] backup metrics implemented for node_exporter + graphana dashboard (space + backup status alerts)
  • [ ] api requests
  • [ ] elasticsearch?

• usage analysis

  • [x] log analyzer goaccess installed and running #300 (https://stats.portal.mardi4nfdi.de) (traefik login data)
  • [ ] most linked/cited/requested data
  • [ ] user queries
  • [ ] ...

• publishing

  • [ ] make some metrics publicly accessible -> https://github.com/MaRDI4NFDI/portal-compose/issues/188
  • [x] #196

• external tools

  • [ ] ~Plausible~: TO BE DROPPED in favor of GoAccess
  • [ ] ~https://github.com/MaRDI4NFDI/T5ProjectManagement/issues/126~
  • [x] ~#280 ~
  • [ ] ~#278 ~

Metrics for our own tools:

• Importer: https://github.com/MaRDI4NFDI/docker-importer/issues/55
• https://github.com/MaRDI4NFDI/portal-compose/issues/313

see https://grafana.wikimedia.org/ for inspiration very rough statistics can be found here: https://portal.mardi4nfdi.de/wiki/Special:Statistics

[Hyper-Node]

Olaf: (für Bibliothekare) Subscription Zugriffszahlen, Wieviele Nutzer/Institutionen sind dabei (nach Ländern), DSGVO abwägen. Johannes (Idee): Heatmaps, weg durch die Dokumente per Nutzer

[dajuno]

https://plausible.io/ looks pretty nice actually, especially from a privacy perspective. quote "P.S. We're completely independent, self-funded, bootstrapped and debt-free. We're not interested in raising funds or taking investment. We choose the subscription business model rather than surveillance capitalism." && "No cookies and fully compliant with GDPR, CCPA and PECR. Made and hosted in the EU, powered by European-owned cloud infrastructure" && open source self-hosted version is completely free

[dajuno]

as suggested by Tim, a log analyzer might be a more adequate choice, since we have access to the apache logs. hence, installed goaccess.io

[physikerwelt]

I would like to give mamoto a shot. In theory the main problem will be to tell mw the IP of traefik so that it trusts it's XFF IP forward headers. https://m.mediawiki.org/wiki/Manual:$wgCdnServers Maybe we need a persistent IP if the server name does not work. Otherwise it seems straight forward. See https://github.com/matomo-org/docker and https://m.mediawiki.org/wiki/Extension:MatomoAnalytics