NethServer / dev

NethServer issue tracker
https://github.com/NethServer/dev/issues
63 stars 18 forks source link

Cloud-init overwrites /etc/hosts #6993

Closed DavidePrincipi closed 1 month ago

DavidePrincipi commented 3 months ago

Some cloud providers configure Cloud-init to overwrite /etc/hosts at every boot. When the leader node is rebooted it is shown offline.

Image

Issue origin and workaround

NS8 actions create-cluster, join-cluster, and promote-node modify /etc/hosts. However those changes can be overwritten during reboot if Cloud-init is configured to expand a template for /etc/hosts.

Manually add /etc/hosts entries to workaround the problem:

127.0.0.1 cluster-leader
10.5.4.1 cluster-localnode

Steps to reproduce

Expected behavior

After rebooting the node is online.

Actual behavior

The node remains offline.

Components

See also

Cloud init docs:

Discussions:

  1. https://mattermost.nethesis.it/nethesis/pl/eiw6k7brefft9me1h7hyxfwk5y
  2. https://mattermost.nethesis.it/nethesis/pl/r56u6uyeitg1xmwx87s6d9rmoh
  3. https://mattermost.nethesis.it/nethesis/pl/ycb5ne5kot8n5dutmd8cy67uuh

Thanks to @nrauso @lucagasparini

DavidePrincipi commented 1 month ago

Test case

Prepare cloud-init configuration files.

mkdir cloud-init
cd cloud-init
echo "instance-id: $(uuidgen)" > meta-data
cat - >user-data <<'EOF'
#cloud-config
hostname: ns8-1
manage_etc_hosts: true
fqdn: ns8-1.ci.test
#ssh_authorized_keys:
#  - "ssh-rsa REPLACE WITH YOUR PUBLIC KEY" 
EOF

# generate the ISO file
rm -f cloud-init.iso
genisoimage -output cloud-init.iso -volid cidata -joliet -rock user-data meta-data

From VM hypervisor, select the cloud-init.iso image as CD-ROM disk image. Boot the NS8 .qcow2 image with cloud-init CD-ROM image.

When boot completes access NS8 console and run this command:

sed -i.orig -r '/-\s*update_etc_hosts/ d ; /-\s*update_hostname/ d' /etc/cloud/cloud.cfg

Complete NS8 setup with a create-cluster.

Power off the VM.

Modify user-data with a different host name, and generate the ISO again.

Start the VM with the new ISO.

Check that /etc/hostname, hostname -f, /etc/hosts didn't change.

Verify cloud-init activity in /var/log/cloud-init.log. For example:

grep -E 'update_hostname|/etc/hosts' /var/log/cloud-init.log 
DavidePrincipi commented 1 month ago

In core 2.9.4-dev.4

DavidePrincipi commented 1 month ago

Released in https://github.com/NethServer/ns8-core/releases/tag/2.9.4