Week 01 routine - Githubissues

kiwixbot commented 9 months ago

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[x] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal (look after last week data)
[x] Mirrorbrain dashboard is normal
[x] There is no abnormal behaviors on cluster resources consumption

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

rgaudin commented 9 months ago

Filesystem      Size  Used Avail Use% Mounted on
bastion        37G  8.1G   27G  23% /
stats        233G   96G  126G  44% /
services        456G  210G  223G  49% /
storage         33T   19T   13T  62% /

restarted cms
mirrorbrain major upgrade to 5.0.0: we faced an issue in that we cant create descending indexex and thus I manually created both index manually (without specifying DESC)

ADD INDEX `index_idsite_idvisitor_time` (`idsite`, `idvisitor`, `visit_last_action_time`);
ALTER TABLE `piwik_log_visit` ADD INDEX `index_idsite_idvisitor_time` (`idsite`, `idvisitor`, `visit_last_action_time`);

; The two requests were originally:
ADD INDEX `index_idsite_idvisitor_time` (`idsite`, `idvisitor`, `visit_last_action_time` DESC);
ALTER TABLE `piwik_log_visit` ADD INDEX `index_idsite_idvisitor_time` (`idsite`, `idvisitor`, `visit_last_action_time` DESC);

zimit

no bottleneck but a lot of requests
(2) Connection refused (Python) to Facebook
(1) certificate verify failed: self-signed certificate (Python)
(1) 403 Client Error: Blocked (Python) for reddit
(1) timeout of 90000 ms
(1) 400 Client Error: Bad Request (Python): URL does return HTTP 400
(1) Name or service not known (Python) : domain does not exist
(1) 404 Client Error (Python) but URL is OK
(1) 404 Client Error (Python): URL does return HTTP 404
(1) 403 Client Error: Forbidden (Python) : URL does return HTTP 403
(1) Unable to find WARC record for main page: seed URL has a tracker query param that might have been removed automatically by the browser. Should be investigated