Month 10 2024 routine - Githubissues

kiwixbot commented 1 month ago

Worker system upgrades

Find a maintenance window for the service:

wait for no more ongoing task on imager then put it in maintenance
run zimfarm stop manager on zimfarm worker

Then:

apt update && apt upgrade

[x] imager-worker
[x] ondemand
[x] sisyphus

Note: this is an automatic reminder intended for the assignee(s).

Zimfarms

[x] Check that we did not unintentionaly kept some secrets in URL output by the farm: this grafana query should return only harmless link (or no logs at all).

rgaudin commented 1 month ago

Done imager-worker

benoit74 commented 1 month ago

Done sisyphus and ondemand.

I also rebooted the machines (just to check everything is fine once rebooted).

Reboot was fine on ondemand but sisyphus refused to start again. Scaleway contacted by phone, they are investigating and will open a ticket for us soon. Probably server has to be changed.

Checked Zimfarm logs OK.

benoit74 commented 1 month ago

See https://github.com/kiwix/operations/issues/278 for sisyphus details.

kiwix / operations

Month 10 2024 routine #274

Worker system upgrades

Zimfarms