installing kubernetes sysext on azure doesn't autostart kubelet

[jepio]

Description

Kubelet is not autostarted after a reboot. In the merged /usr ls /usr/lib/systemd/multi-user.target.d shows that only the waagent conf is present. Manually mounting the kubernetes sysext confirms that it has a config that should uphold it.

Impact

[ 1 sentence detailing the impact this bug is creating for you ]

Environment and steps to reproduce

1. Set-up: [ describe the environment Flatcar/Lokomotive/Nebraska etc was running in when encountering the bug; Platform etc. ]
2. Task: [ describe the task performing when encountering the bug ]
3. Action(s): [ sequence of actions that triggered the bug, see example below ] a. [ requested the start of a new pod or container ] b. [ container image downloaded ]
4. Error: [describe the error that was triggered]

Expected behavior

[ describe what you expected to happen at 4. above but instead got an error ]

Additional information

Please add any information here that does not fit the above format.

[pothos]

It's a bug in the OEM sysext. The way the directory usr/lib/systemd/system/multi-user.target.d got created led to it being opaque, i.e., attr -R -g overlay.opaque says y which means that lower layer directories get ignored.

[pothos]

This is a problem for many directories:

sudo systemd-dissect --with  ./oem-azure.raw find usr/ -xdev -type d -exec sh -c 'if [ "$(attr -R -q -g overlay.opaque {} 2>/dev/null)" = y ]; then echo {}; fi' \;
usr/lib/python
usr/lib/python-exec
usr/lib/python3.11
usr/lib/systemd/system/multi-user.target.d
usr/share/waagent
$ sudo systemd-dissect --with oem-vmware.raw find usr/ -xdev -type d -exec sh -c 'if [ "$(attr -R -q -g overlay.opaque {} 2>/dev/null)" = y ]; then echo {}; fi' \;
usr/lib/systemd/system/multi-user.target.d
usr/lib/systemd/system/vgauthd.service.d
usr/lib/systemd/system/vmtoolsd.service.d
usr/lib64/open-vm-tools
usr/share/flatcar/oem-vmware/vmware-tools
usr/share/open-vm-tools

In theory it's correct but since we want to have multiple overlays merge their contents, we should postprocess this.

[pothos]

Turns out it was also masking the containerd upholds we ship through the internal containerd sysext