Week 39 routine - Githubissues

kiwix / operations

Kiwix Kubernetes Cluster

6 stars 0 forks source link

Infra

[x] Check nodes free space
```
df -h / && df -h /data
```
[x] Nodes and worker system upgrades
```
apt update && apt upgrade
```
[x] Ensure all borg repositories are being updated
[x] Check Pod errors
```
k get pods -A -o wide|grep Error
```

[x] Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

[ ] Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER|jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions|jq ".versions[].name"

[x] Upgrade k8s if applicable

Stats

[x] Ensure download.kiwix.org stats are being recorded
[x] Check whether matomo should be upgraded

Grafana

[x] Alert list is normal
[x] Zimfarm dashboard is normal (look after last week data)

Projects

[x] UptimeRobot has no alert
[x] youzim.it backlog is reasonable
[x] No systematic failure in tasks.
[x] PRs awaiting your review

Security

[x] Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

Filesystem	Size	Used	Avail	Use%
bastion	37G	5.2G	30G	15%
stats	233G	90G	132G	41%
services	456G	186G	247G	43%
storage	33T	19T	13T	59%

Filesystem

Size

Used

Avail

Use%

bastion

37G

5.2G

30G

15%

stats

233G

90G

132G

41%

services

456G

186G

247G

43%

storage

33T

19T

13T

59%

{"timestamp":"2023-09-20T03:01:59.247Z","logLevel":"error","context":"worker","message":"Page Crashed","details":{"type":"exception","message":"Page crashed!","stack":"Error: Page crashed!\n at #onTargetCrashed (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/Page.js:284:28)\n at file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/Page.js:153:41\n at file:///app/node_modules/puppeteer-core/lib/esm/third_party/mitt/index.js:1:248\n at Array.map (<anonymous>)\n at Object.emit (file:///app/node_modules/puppeteer-core/lib/esm/third_party/mitt/index.js:1:232)\n at CDPSessionImpl.emit (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:82:22)\n at CDPSessionImpl._onMessage (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/Connection.js:425:18)\n at Connection.onMessage (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/Connection.js:255:25)\n at WebSocket.<anonymous> (file:///app/node_modules/puppeteer-core/lib/esm/puppeteer/common/NodeWebSocketTransport.js:46:32)\n at callListener (/app/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)","page":"https://plotly.com/python/3d-surface-plots/","workerid":0}}