Increase K3s cluster stability

[backstreetkiwi]

During our research for https://github.com/stackabletech/t2/issues/368, we tried to experiment with 2 long-running K3s clusters. Unfortunately, they did not really run for a long time but crashed pretty soon.

Symptoms:

• Nearly every test ran into timeouts
• The orchestrator as well as some of the agent-nodes were not reachable, even with SSH
• The Hetzner graphs showed that the CPU power was hitting its (physical) limit

The post-mortem analysis was not so easy because the journals were gone after reboot (see https://github.com/stackabletech/infrastructure/issues/59)

In this task, we should:

• set the K3s cluster to keep the journals (Storage=persistent, see https://www.freedesktop.org/software/systemd/man/journald.conf.html)
• Try to set the cluster under some load by running some tests
• Analyse what has been going wrong and try to come up with ways to fix it.

[backstreetkiwi]

We switched to managed K8s completely, maybe also because of problems like this. I guess we could close this issue as well, @Jimvin ?