Closed mr-karan closed 1 month ago
Hi @mr-karan! It's a little strange that you're finding a regression here, as Nomad doesn't load the kernel module on its own. That's actually a known issue https://github.com/hashicorp/nomad/issues/10902 (and also https://github.com/hashicorp/nomad/issues/17311, sort of). But we also have https://github.com/hashicorp/nomad/issues/23523 open recently where someone wasn't seeing network fingerprinting happen correctly but seems to think it's Docker-related.
Is there any chance you've updated other infrastructure components other than Nomad when seeing this regression, @mr-karan? Specifically kernel, distro, Docker version, etc.?
Ah yes, indeed we upgraded to Ubuntu Minimal 24.04 (we were on 22.04 Minimal in 1.6.1). I'll try to reproduce this in a VM and get back. Thanks for taking a look!
Similar issue reported here: https://github.com/hashicorp/nomad/issues/23700
We'll need to update install documentation on this, as lots of folks are going to get caught out by it unfortunately.
Docs are being updated in #23707
Nomad version
Output from
nomad version
: Nomad v1.7.7Operating system and Environment details
OS: Linux (AWS EC2) Kernel version: 6.8.0-1008-aws Environment: AWS EC2 instance
Issue
After upgrading Nomad from 1.6.1 to 1.7.7, the
bridge
network is not functioning. The Nomad agent logs indicate that thebridge
module is not found or loaded in the system kernel.Reproduction steps
Expected Result
The
bridge
network functions correctly without any errors, similar to the behavior seen in version 1.6.1.Actual Result
The following errors are observed in Nomad client logs indicating issues with the
bridge
module:failed to find /sys/module/bridge: no such file or directory
module bridge not in /proc/modules
module bridge not in /lib/modules/6.8.0-1008-aws/modules.builtin
module bridge not in /lib/modules/6.8.0-1008-aws/modules.dep
Nomad Client logs (if appropriate)
Note
I fixed it by manually loading the module with
sudo modprobe bridge
. After doing this, I restarted Nomad and spawned my job which hadnetwork.mode="bridge"
and it worked fine. I think this is a regression from 1.6->1.7 upgrade.