Open char8 opened 1 month ago
updated the ticket after tracing the behaviour change to #386. We've pinned ourselves to 1:20240528.00-g1
for the moment. It would seem that the guest agent should regenerate the netplan configs on boot based on the ifname; but we're not seeing it do that.
We've further identified the reason for the network interface name change to be the addition of local SSDs on the VM running the image (the VM running packer to build the image lacked NVMEs). The NVMe takes PCIe slot ID 4, pushing the NIC to PCIe slot 5, resulting in the change from ens4 -> ens5 based on systemd's naming scheme.
We're looking into this issue. Does instance have any networking or its completely broken?
Networking is completely broken. Cannot SSH or anything
We have the same issue.
We have a package built from a previous stable state (with a newer version number) and are finishing up upgrade testing from the version with the issue to the previous stable package. We hope to release it shortly, and will update this bug with an ETA as soon as we have it.
We also have broken networking after guest-agent updated on our system.
We're getting logs like:
Jul 17 13:19:34 app1-staging-app google_guest_agent[1683]: Rolling back systemd-networkd
Jul 17 13:19:34 app1-staging-app google_guest_agent[1683]: rolling back changes for systemd-networkd
It seems related to this:
Jul 17 12:55:30 app1-staging-app google_guest_agent[1665]: Setting up NetworkManager
Jul 17 12:55:30 app1-staging-app NetworkManager[779]: <warn> [1721220930.0963] keyfile: guest-agent: invalid setting name 'guest-agent'
Jul 17 12:55:30 app1-staging-app s3fs[1037]: ### retrying...
We're running a Rocky 8 system and NetworkManager doesn't seem to understand the config that google-guest-agent is placing at /etc/NetworkManager/system-connections/google-guest-agent-eth0.nmconnection. The version of NetworkManager on this system is NetworkManager-1.40.16-15.el8_9.x86_64
. We have a Rocky 9 system where this isn't a problem, but that one has a newer version of NetworkManager which does understand the config: NetworkManager-1.46.0-8.el9_4.x86_64
The team will be releasing the new (old) version this morning.
Thanks for bringing up the issues you're seeing. We have rolled back the changes and released another agent 20240716.00
version.
Hi @mikehardenize , we believe this might be a different issue. Can you provide reproduction steps and what functionality is not working that you're expecting to work? The log messages you have posted are expected, the agent will log Rolling back X
for every network management software that it's not going to configure, and it prints Setting up NetworkManager
with no error log after indicating successfully writing NetworkManger config files.
The NetworkManger warnings indicate that it doesn't understand the guest-agent
ini key but this is expected and as far as we can tell the configuration file is otherwise applied and the NIC is activated and configured correctly. Is there some functionality breakage you're seeing? We can't reproduce the broken networking on Rocky Linux 8.
I gotta say it is scary as heck to see pull requests like this
We're lucky we run a fleet of canary instances and caught this early before it made it to our other instances.
I won't be dogmatic about the solution, but, given the total littany of issues we're run into with Google cloud, this is one of the many nails in the coffin of "This cannot be trusted for production infrastructure"
For anyone still facing these issues, follow these steps to work around and update the guest-agent
. Similar steps can be found here.
sdb
, the command may look like the following:
sudo mkdir /mnt/data && sudo mount /dev/sdb1 /mnt/data
/mnt/data/etc/netplan/20-google-guest-agent-ethernet.yaml
./mnt/data/etc/netplan/90-default.yaml
, and paste the following contents inside:
network:
version: 2
ethernets:
all-en:
match:
name: en*
dhcp4: true
dhcp4-overrides:
use-domains: true
dhcp6: true
dhcp6-overrides:
use-domains: true
all-eth:
match:
name: eth*
dhcp4: true
dhcp4-overrides:
use-domains: true
dhcp6: true
dhcp6-overrides:
use-domains: true
sudo umount /dev/sdb1
Hi @mikehardenize , we believe this might be a different issue. Can you provide reproduction steps and what functionality is not working that you're expecting to work?
It's a Rocky Linux 8 system. We are using wireguard via systemd-networkd. After the upgrade of guest-agent, a default route is added for that wireguard gateway ip. This breaks our networking. The default route wasn't added prior to the upgrade.
We pulled in a new release of the guest agent (
1:20240701.00-g1
) incorporating https://github.com/GoogleCloudPlatform/guest-agent/pull/396 and https://github.com/GoogleCloudPlatform/guest-agent/pull/386 during a packer build of a new VM image.This guest agent now writes a file
/etc/netplan/20-google-guest-agent-ethernet.yaml
with the contents:vs. the previous default
/etc/netplan/90-default.yaml
the interface on the build instance is
ens4
and the20-google-guest-agent-ethernet.yaml
file hardcodes that interface name into the image.When the Image is run on a new VM, if that VM has a different network interface name (eg: we're seeing
ens5
on some VMs), the network interface fails to come up since the declaration in the condig file is missing. This effectively breaks networking on the box as theens5
interface is never brought up because/run/systemd/network/10-netplan-all-en.network
is missing.run/systemd/network/10-netplan-all-en.network
and/etc/netplan/90-default.yaml
is missing post upgrade.ens4
) to the VMs on our managed instance group (which come up withens5
).We're running the
debian-cloud/debian-12
image with:Post reboot; the guest agent is crashing because it can't reach the metadata API (since ens5 is not up), so it wouldn't be able to presumably re-generate the config for the new interface name.