Opening and closing this just as a tip in case anyone else runs into this.
Symptoms:
Sometimes (10-20%?) on reboot, our EFS mount will not come up. /var/log/amazon/efs/mount.log showed ERROR - Failed to mount fs-0123456789abcdef because the network was not yet available, add "_netdev" to your mount options. We are already using _netdev. I found that error message here, in a check that network.target is up. When the mount failed, network.target is not up.
Cause:
This is on Debian 11. We've had a standard list of packages we've been installing for ages through many distros that includes fcoe-utils. As discussed here and resolved in v1.0.34-3, some versions of fcoe-utils create a circular dependency between itself, lldpad, and network.target. Sure enough that's exactly what we saw, systemd \~randomly~ choosing a job to delete to break the circle.
On reboots that result in EFS mounting properly, journalctl shows like systemd[1]: network.target: Job fcoe-utils.service/start deleted to break ordering cycle starting with network.target/start
On reboots that result in EFS failing to mount, journalctl shows like systemd[1]: lldpad.service: Job network.target/start deleted to break ordering cycle starting with lldpad.service/start
Workaround
For us, simply uninstalling, disabling or masking the fcoe-utils service is sufficient. We don't need it.
Opening and closing this just as a tip in case anyone else runs into this.
Symptoms:
Sometimes (10-20%?) on reboot, our EFS mount will not come up.
/var/log/amazon/efs/mount.logshowedERROR - Failed to mount fs-0123456789abcdef because the network was not yet available, add "_netdev" to your mount options. We are already using_netdev. I found that error message here, in a check that network.target is up. When the mount failed, network.target is not up.Cause:
This is on Debian 11. We've had a standard list of packages we've been installing for ages through many distros that includes
fcoe-utils. As discussed here and resolved in v1.0.34-3, some versions offcoe-utilscreate a circular dependency between itself,lldpad, andnetwork.target. Sure enough that's exactly what we saw, systemd \~randomly~ choosing a job to delete to break the circle.systemd[1]: network.target: Job fcoe-utils.service/start deleted to break ordering cycle starting with network.target/startsystemd[1]: lldpad.service: Job network.target/start deleted to break ordering cycle starting with lldpad.service/startWorkaround
For us, simply uninstalling, disabling or masking the fcoe-utils service is sufficient. We don't need it.