Week 44 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [x] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal (look after last week data)
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] youzim.it backlog is reasonable
• [x] No systematic failure in tasks.
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Filesystem	Size	Used	Avail	Use%	Change
bastion	37G	7.3G	28G	21%	+1.9G ; +5%
stats	233G	90G	132G	41%	-
services	456G	198G	235G	46%	+7G ; +1%
storage	33T	19T	13T	60%	- ; +1%

No package update

dev-library has restarted again today

Spike of 502 errors on Zimfarm API on 31th October ; some 500 errors today due to system unavailability

Mirrorbrain update DB experienced lots of slowness (up to 3h40 to run one single update) from 30th october to 1st november, issue seems to be solved since few hours

zimit failures

3x 403 error (python) 3x 403 error (crawler) 1x 405 Error (crawler) 24x “Description is too long.” 1x 500 Error (python) 1x 520 Error (python) 2x 406 Error (python) 2x timeout (crawler) 3x CERTIFICATE_VERIFY_FAILED due to bad certificates 2x DH_KEY_TOO_SMALL (same website) 2x corrupted size vs. prev_size (https://farm.youzim.it/pipeline/bd381e5a-fccc-4a91-abfc-25681d61d2a9 and https://farm.youzim.it/pipeline/37614f25-7430-434d-8300-a3c81339bbd5) 1x --depth: invalid int value: '' 2x Time threshold reached 1x RuntimeError: Unknown exception (https://farm.youzim.it/pipeline/e97bcf8b-f269-40a8-9f83-1af8a3ff52d0) 1x worker error with Docker API (https://farm.youzim.it/pipeline/9c16585f-e7f7-480c-aab2-0b8d83337436)