Week 14 routine - Githubissues

kiwixbot commented 3 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

benoit74 commented 2 months ago

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	`/`	37G	8.9G	27G	26%	-
stats	`/`	233G	100G	122G	45%	+3G, +1%
services	`/`	456G	235G	1983G	55%	+5G, +1%
storage	`/`	33T	20T	12T	65%	+1T, +4%
demo	`/`	916G	182G	688G	21%	?
imager-worker	`/`	1.9T	816G	974G	46%	?
sisyphus	`/`	233G	11G	320G	5%	?
ondemand	`/`	25G	9.5G	15G	41%	?
ondemand	`/data`	216G	50G	155G	25%	?

Many apt packages updated on machines.

Multiple backups are too old, last invocation just failed due to Bitwarden issue, everything is back to normal so I triggered the jobs manually to have a backup now.

Zimit

90000 ms navigation timeout
- https://www.talkorigins.org
- http://amdxer.com/ambc/
Python 404
- https://image13.photobiz.com/10309/ => also a 404 on my machine
- https://ruthtoledofotografia.es/web/wp-content/uploads/2023/11/ => also a 404 on my machine
- https://www.valpaint.it/img/public/new/PRODOTTI/ARTECO7/ => working ok on my machine
Python 403
- https://antoniojeffersonstudio.pixieset.com/dreboni/ => working ok on my machine
- https://minecraft.wiki/ => again and again ...
- https://www.infowars.com => working ok on my machine
- 2x https://old.reddit.com/r/Piracy/wiki/megathread => working ok on my machine
- 2x https://community.openvpn.net => working ok on my machine
- http://sportsmed.in/wp-content/uploads/ => also ko on my machine
- https://pf2.huijiwiki.com => working ok on my machine
Python DNS problem
- https://zlarrm.ekiwi.com => also a DNS issue on my machine
Python connection timeout
- https://cipher-compendium.com/series/booster-series/B01 => ok on my machine
- https://www.granikos.com.tr/ => ok on my machine
https://github.com/openzim/warc2zim/issues/203
- https://edisciplinas.usp.br/pluginfile.php/4557662/mod_resource/content/1/CRC%20Handbook%20of%20Chemistry%20and%20Physics%2095th%20Edition.pdf
few WP1 issues (key to S3 file with article list does not exists ... most probably linked to the file being already deleted since the task never achieved to start in due time)