Week 42 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• [x] create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• [x] run systematically the upgrade on bastion, stats, services, storage, demo, mirrors-qa nodes
• [x] check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• [x] Ensure all borg repositories are being updated

k8s cluster

• [x] Check Pod errors or in CrashLoopBackoff

  k get pods -A -o wide|grep -E 'Error|Crash'

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [ ] Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Jobs durations dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption
• [x] All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• [x] Analyze zimit failed tasks
• [x] PRs awaiting your review

Security

• [ ] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	16G	20G	44%	-
stats	/	233G	111G	110G	51%	+1G
services	/	456G	329G	110G	76%	+6G
storage	/	147G	44G	96G	28%	+5G
storage	/data	30T	14T	15T	48%	-
imager-worker	/	1.9T	549G	1.3T	31%	don't care
sisyphus	/	233G	9.6G	211G	-	don't care
ondemand	/	25G	9.7G	14G	42%	-
ondemand	/data	216G	2.2G	203G	2%	don't care
mirrors-qa	/	38G	4.4G	32G	13%	-
demo	/	40G	9.3G	28G	26%	+0.3G
demo	/data	1.8T	1.2T	472G	72%	don't care

For some reasons, it looks like sisyphus apt packages are outdated despite being freshly installed last week. I probably missed something. Will be updated next month, we have too much tasks in youzim.it pipe to stop the worker now.

k8s cluster

• Zimfarm watcher (to periodically mirror StackExchange) is down due to archive.org downtime

• 2 successives dev library generation failed: https://github.com/kiwix/operations/issues/283

Grafana

• https://github.com/kiwix/operations/issues/284

Projects

• Unused Signings: 973

@Jaifroid @rgaudin why did it dropped from 1035 to 973 in one week, is it normal to consume >50 signins so fast?

Zimit

• Pipe is mostly empty (2 running tasks)
• (1) net::ERR_ABORTED (Seed Page Load Failed)
• (1) net::ERR_HTTP2_PROTOCOL_ERROR (Seed Page Load Failed)
• (1) net::ERR_TIMED_OUT (Seed Page Load Failed)
• (1) net::ERR_EMPTY_RESPONSE (Seed Page Load Failed)
• (2) net::ERR_NAME_NOT_RESOLVED (Seed Page Load Failed)
• (1) net::ERR_CONNECTION_REFUSED (Seed Page Load Failed)
• (8) Navigation timeout of 90000 ms exceeded (Seed Page Load Failed)
• (9) Browser disconnected (crashed?), interrupting crawl
• (3) https://github.com/openzim/warc2zim/issues/331: unicode, iso-8859
• (2) ValueError: Title is too long.
• (1) Invalid seed specified, aborting crawl.

[rgaudin]

On signatures: https://github.com/kiwix/kiwix-js-pwa/releases

[Jaifroid]

@benoit74 Yes, each Windows Electron package has to be signed around three times, including signing the uninstaller, the installer and the underlying executable, and there are several packages (appx, appxbundle, ARM64, x64, ia32, and a portable exe). But we did a calculation and with monthly releases of the three apps, plus Kiwix Desktop releases, there should still be ample left on our current plan. I did also have an issue this time with a package that had the wrong MS Store Publisher DisplayName, which was rejected, and had to rerun, but I know the cause and that's fixed for all future releases.

So, we should be fine. I don't sign nightlies anymore.