Self reconcile

[mwindower]

• self-reconcile firewall-controller binary based on the version given in the firewall spec - closes #57
• proper use of release-drafter
• release firewall-controller also as binary together with SHA256 checksum
• controller-gen@v0.4.0

needs adoption in gepm, cloud-api and cloudctl

[majst01]

Two questions comes in mind:

• can the binary be replaced "in-place", during my work in the https://github.com/metal-stack/updater i found that metalctl for example cant be overwritten during it is running.
• What if the new binary is able to run, but wont be able to connect to the k8s cluster for example, then we end up with a dead firewall-controller, and this will happen through the whole fleet once the version is updated. I dont have a good answer to this except that we introduce a second daemon which checks if firewall-controller is running smoothly after a upgrade and if not replace it with the last good known version.

Idea for the watchdog:

• in the firewall-controller at the end of the reconcile loop, the actual version is written to a known place /etc/metal/firewall-controller-status.yaml
• if a knew version is specified, right at the beginning this version is also noted there maybe like so:

  ---
  lastversion: v0.2.0
  actualversion: v0.2.0
  lastreconcile: 2020-11-24:09-10.34
  nextversion: v0.2.1

• the watchdog will check every 10sec if nextversion == actualversion and lastreconcile is not older than a reasonable amount of time.
• TODO: now the watchdog needs to modify the desired version and reset the binary to the old one ?

Maybe this is to difficult and error-prone

[mwindower]

• I wrote a small test program to show that "in-place" update works: https://gist.github.com/mwindower/c0d28eda63f4a2310696680d98855fe0
• your second concern was about a setup with a new firewall-controller that could possibly not communicate with older k8s api server versions?

[majst01]

No second concern is howto handle a situation if the update fails for whatever reason, how do we come back to the previous known stable state.

[mwindower]

For normal errors like download / checksum errors, the controller does a retry already.

If os.Rename fails from the temp location of the new binary to the real location fails, we should have a backup of the old version? - this is not yet covered.

[majst01]

Or if we specify a broken version, all firewalls will be borked, this is my concern

[mwindower]

This is not an auto-update to the latest version. We would definitely update firewall-controller versions in a controlled manner... meaning setting it with cloudctl cluster update

[majst01]

OK, lets go for it, agreed