Week 42 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal (look after last week data)

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [ ] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Filesystem	Size	Used	Avail	Use%	Change
bastion	37G	5.3G	30G	16%	+0.1G ; +1%
stats	233G	90G	132G	41%	-
services	456G	189G	244G	44%	-
storage	33T	19T	13T	59%	- ; -1%

3 updates on kstorage, kservices et kstats:

• curl (7.74.0-1.3+deb11u10) over (7.74.0-1.3+deb11u9)
• libcurl4:amd64 (7.74.0-1.3+deb11u10) over (7.74.0-1.3+deb11u9)
• libcurl3-gnutls:amd64 (7.74.0-1.3+deb11u10) over (7.74.0-1.3+deb11u9)

@rgaudin cardshop has not had a single visit recorded on stats.kiwix.org for the last 4 days, is it possible or an issue?

Some series of 502 errors on api.farm.openzim.org, looks like burst of traffic which overwhelmed the API. To monitor if this continue to happen.

In youzim.it, multiple jobs finishing with seed page load errors (403/404) while the website looks indeed ok (difficult to consider that all of them have been down when the job started).

[rgaudin]

@rgaudin cardshop has not had a single visit recorded on stats.kiwix.org for the last 4 days, is it possible or an issue?

I'll look into it. It's possible, last order was on October 6th. I know the service is running because I use it… but I block the matomo JS. Given we have a parallel system in testing I can't exclude a side effect.

In youzim.it, multiple jobs finishing with seed page load errors (403/404) while the website looks indeed ok (difficult to consider that all of them have been down when the job started).

We have a ticket for that on zimit. Make sure you check the status code as well ; From last week, some URLs were returning what looks like proper HTML content but status code was 404. 403 might be related to anti-crawling services.