Survive reboot gracefully

[brokenpylons]

To recover form a reboot the following steps had to be taken:

• The network had to be recreated using terraform
• All domains had to be started manually
• Acme timers had to be restarted manually

The main problem is that the domains don't get the correct ip and hostname form DHCP, Recreating the network with terraform works, however after a while it stops working for some reason. Just starting the network with virsh doesn't work at all.

A possible fix is to bake the hostnames into the images again.

[brokenpylons]

The network_interface part of the configuration actually sets the dhcp record in the network. The current solution is to set all three explicitly (also the MAC), and the hosts entries for the DNS are no longer needed.

[brokenpylons]

The dhcp records aren't permanent, that is why the configuration gets reset after reboot (net-update). The final solution was to replace Terraform with a small bash script. Now we can edit the network configuration directly and everything works as expected. The machines have to be rebooted after the network configuration is changed.