Week 24 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [ ] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [ ] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Analyze zimit failed tasks
• [ ] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	12G	24G	33%	-
stats	/	233G	102G	120G	46%	-1G
services	/	456G	278G	155G	65%	+3G
storage	/	33T	21T	10T	68%	-
demo	/	916G	126G	744G	15%	don't care
imager-worker	/	1.9T	831G	959G	47%	don't care
sisyphus	/	233G	14G	207G	7%	-
ondemand	/	25G	9.8G	14G	42%	-
ondemand	/data	216G	202G	205G	1%	+144G, -2%

misc

• ondemand zimfarm worker was reconfigured to use more disk space
• docker restart for imager worker (no task) and zimit worker: broke 3 tasks: 1a25f, 8b077 and f60f2
• zimfarm watcher restarted (Connection reset by peer, not in S3 so retrieving XML I guess)
• There's a matomo update (5.0.3 ➡️ 5.1.0) without major DB change. Update wasn't applied because our (kiwix/container-images) image cant pull upstream image as matomo did not request image at docker. Please check my PR next week https://github.com/docker-library/official-images/pull/16958
• unable to load grafana dashboards… grafana experiencing issues?

zimit

• backlog is slowly shrinking: 377 pending
• 2 x ConnectTimeoutError
• 22 x 403 Client Error: Forbidden
• 2 x 404 Client Error: Not Found
• 1 x Invalid WARC record (fixed in zimit2 browsertrix)
• 1 x RemoteDisconnected
• 1 x 403 Client Error: Blocked
• 1 x 401 Client Error: Authorization Required
• 2 x SSLCertVerificationError
• 1 x NameResolutionError
• 1 x Connection refused
• 1 x Invalid Seed Url
• 1 x Page Load Timeout, failing crawl
• 1 x Read timed out
• 1 x No address associated with hostname
• 1 x 401 Client Error: Unauthorized

[rgaudin]

Opened https://github.com/webrecorder/browsertrix-crawler/issues/598 for that Invalid Seed Url one