Week 38 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• [x] create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• [x] run systematically the upgrade on bastion, stats, services, storage, demo nodes
• [x] check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• [x] Ensure all borg repositories are being updated

k8s cluster

• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [ ] Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Jobs durations dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption
• [x] All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• [ ] Analyze zimit failed tasks
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	15G	21G	42%	-
stats	/	233G	110G	111G	50%	+1G
services	/	456G	315G	118G	73%	+4G
storage	/	147G	23G	117G	16%	+5G
storage	/data	30T	18T	11T	63%	+1T
imager-worker	/	1.9T	568G	1.2T	32%	don't care
sisyphus	/	233G	23G	198G	11%	don't care
ondemand	/	25G	9.7G	14G	42%	-
ondemand	/data	216G	204M	205G	1%	don't care
mirrors-qa	/	38G	3.8G	32G	11%	-1G
demo	/	40G	9.5G	29G	23%	-1G
demo	/data	1.8T	925G	739G	56%	don't care

misc

On Thursday 2024-09-12 at 10:25 a couple of pods restarted:

• zimfarm.watcher-deployment-54cf77666f-tgfjw
• grafana.grafana-k8s-monitoring-kube-state-metrics-78c4cb9dd7-g76w5
• cert-manager.cert-manager-fdd97855b-m8stj

cert-manager is known to restart from time to time. The fact that at the same moment two other ones were restarted is interesting.

---

Jobs generation duration between 2024-09-13 and 2024-09-14 were significantly longer

---

Cloud Signing: 1,035 left

---

Uptime robot incorrectly displays Kiwix Wiki as down (for a year!) but old dashboard is OK. I reported the issue to them (via their Chat)

zimit

• Backlog has 7 tasks
• Two slots taken bu 15d long tasks. @benoit74 why didn't you cancel those during last week's routine? Containers are not present.
• We have a few recipes that were not deleted. https://github.com/kiwix/operations/issues/256
• 3 x https://github.com/openzim/warc2zim/issues/331: u, 3Diso-8859-1
• 1 x Navigation timeout of 90000 ms exceeded
• 1 x https://github.com/webrecorder/browsertrix-crawler/issues/687
• 7 x Seed Page Load Failed
• 4 x https://github.com/openzim/warc2zim/issues/383
• 1 x https://github.com/openzim/warc2zim/issues/394
• 2 x ValueError: Title is too long
• 1 x Invalid values for behaviors

I'm getting a bit concerned about Seed Page Load Failed. A lot of them seems to work fine on my browser. It seems some WAF are triggering this now.

[benoit74]

@benoit74 why didn't you cancel those during last week's routine?

I simply forgot to do it. Probably got interrupted by something else...

I'm getting a bit concerned about Seed Page Load Failed. A lot of them seems to work fine on my browser. It seems some WAF are triggering this now.

This is not something really new, isn't it?

[rgaudin]

This is not something really new, isn't it?

It's zimit2, basically since we removed the python front-test I believe. The problems I see are:

• this is a very wide feedback: there can be many cause for this ; from a slow responding website to an incorrect URL or a WAF (they all seem to raise this now instead of the multiple ways we got before)
• we have no ticket nor documentation about this.

[benoit74]

we have no ticket nor documentation about this.

It is now documented at https://github.com/openzim/zimit/wiki/Frequently-Asked-Questions#the-zim-is-not-created-and-logs-says-seed-page-load-error