Week 40 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal (look after last week data)

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Filesystem	Size	Used	Avail	Use%
bastion	37G	5.2G	30G	15%
stats	233G	90G	132G	41%
services	456G	189G	244G	44%
storage	33T	19T	13T	60%

Pods library-demo-backend-deployment both restarted two days ago.

farm.openzim.org experienced a bunch of 502 errors (no more logs available):

2023-10-01T22:21:03.8069443Z stdout F 100.64.6.28 - - [01/Oct/2023:22:21:03 +0000] "GET /v1/schedules/mali-pour-les-nuls-mp4 HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "85.26.62.218"
2023-10-01T22:21:03.806951153Z stdout F 100.64.6.28 - - [01/Oct/2023:22:21:03 +0000] "GET /v1/schedules/avanti-mathematical-reasoning HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "85.26.62.218"
2023-10-01T22:21:03.807068901Z stdout F 100.64.6.28 - - [01/Oct/2023:22:21:03 +0000] "GET /v1/schedules/avanti-wave-optics HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "85.26.62.218"
2023-10-01T22:21:03.807075279Z stdout F 100.64.6.28 - - [01/Oct/2023:22:21:03 +0000] "GET /v1/schedules/avanti-solid-state HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "85.26.62.218"
2023-10-01T22:21:03.80719722Z stdout F 100.64.6.28 - - [01/Oct/2023:22:21:03 +0000] "GET /v1/schedules/avanti-s-block-elements HTTP/1.1" 502 157 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0" "85.26.62.218"

Youzimit:

• task 3720db44-8d4b-45b2-8613-c7cd4139378e stuck since 5 days (but the person was trying to zim www.vimeo.com ...) => cancelled manually
• backlog is empty 👍
• More issues of https://github.com/openzim/zimit/issues/185#issuecomment-1744348960 (including small variations)
• More issues of https://github.com/webrecorder/browsertrix-crawler/issues/372 (including one variation with another CDN, upstream issue updated)

[rgaudin]

did you forget to close the issue?

[benoit74]

I got interrupted, but did not forget. You are too quick 😆