Week 23 routine

[kiwixbot]

Infra

• [x] Check nodes free space

  df -h / && df -h /data

• [ ] Nodes and worker system upgrades

  apt update && apt upgrade

• [x] Ensure all borg repositories are being updated
• [x] Check Pod errors

  k get pods -A -o wide|grep Error

• [x] Check Pod restarts

  k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• [x] Check if k8s should/could be upgraded

  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
  curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• [x] Upgrade k8s if applicable

Stats

• [x] Ensure download.kiwix.org stats are being recorded
• [x] Check whether matomo should be upgraded

Grafana

• [x] Alert list is normal
• [x] Zimfarm dashboard is normal
• [x] Mirrorbrain dashboard is normal
• [x] There is no abnormal behaviors on cluster resources consumption

Projects

• [x] UptimeRobot has no alert
• [x] zimit backlog is reasonable
• [x] Analyze zimit failed tasks
• [x] PRs awaiting your review

Security

• [x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[benoit74]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	12G	24G	33%	-
stats	/	233G	102G	120G	46%	-1G
services	/	456G	275G	158G	64%	+2G
storage	/	33T	21T	11T	68%	+1%
demo	/	916G	35G	835G	4%	don't care
imager-worker	/	1.9T	865G	925G	50%	don't care
sisyphus	/	233G	14G	207G	7%	-1G
ondemand	/	25G	9.7G	14G	42%	-
ondemand	/data	216G	58G	147G	29%	-5G, -2%

Misc

• Upgrade of linux-libc-dev on most nodes
• StackExchanger watcher regularly restarts after connection reset by peer errors: https://github.com/openzim/zimfarm/issues/980

zimit.kiwix.org

• 3 tasks got stuck (due to usage of zimit2 image, which still had the infinite loop issue at that time), blocking the pipe which is now at 516 pending tasks, oldest one being from 29 days ago
• 5 tasks have been cancelled by the periodic scheduler which observed they took too long
• Python 403 forbidden
  • https://www.gplanet.co.il/
  • https://www.physio-pedia.com
  • https://www.vocabulary.com/
  • https://pzwiki.net
  • https://pixabay.com/
  • https://www.reddit.com/r/ElegooNeptune4/
  • https://www.tudogostoso.com.br
  • https://minecraft.wiki/
  • https://darksouls.wiki.fextralife.com/Dark+Souls+Wiki
  • https://www.msi.com/
  • https://lightnovelbrasil.com/series/a-returners-magic-should-be-special/
  • https://myreadingmanga.info/
  • https://octostera.artstation.com
  • https://centraldeatendimento.totvs.com/hc/pt-br/sections/1500001529881-TOTVS-Backoffice-Linha-Protheus
  • https://awoiaf.westeros.org/index.php
  • https://mangafire.to/read/vinland-saga.zpw
• Python 404 Not Found
  • https://programminginception.com/intro
• Python Connection Timeout
  • http://www.dominiopublico.gov.br/pesquisa/
• Python SSL issues
  • https://write.flossmanuals.net/pure-data/
  • https://crca.ucsd.edu/~msp/techniques/latest/book-html/
  • https://pd-tutorial.com/
• Python connection timeout
  • http://www.dominiopublico.gov.br/pesquisa/
  • https://institutoluizgama.org.br/
• Python 500
  • https://salavirtual.pucrs.br/curso/mba-em-lideranca-gestao-de-equipes-e-produtividade-68/conteudo?item=378909%3A
• Python DNS Name resolution issue:
  • https://www.pt.wikipedia.org
• Crawler page load timeout on https://www.deezer.com/br/channels/podcasts
• Navigation timeout of 90s on https://www.plantapronta.com.br/ and https://www.plantapronta.com.br/plantas-de-casas/
• Corrupted unsorted chunks error at the end of ZIM creation: https://github.com/openzim/warc2zim/issues/295
• Yet another decoding issue on fontawesome-webfont.woff2: https://github.com/openzim/warc2zim/issues/296
• Wrong behavior given: Argument: behaviors, Given: "nopic", Choices: "autoplay", "autofetch", "autoscroll", "siteSpecific" (nopic?)