OPNsense 24.7: "ping: sendto: No buffer space available" with Tor relays and pfatt

[neelchauhan]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

On a heavily loaded OPNsense installation, for instance one running Tor relays, the connection after a few hours freezes up and gives me a ping: sendto: No buffer space available error.

This happens when using the pfatt AT&T Fiber bypass on Frontier FiberOptic which in Connecticut uses 802.1X for non-XGS-PON.

To Reproduce

Steps to reproduce the behavior:

1. Set up an OPNsense box with the pfatt bypass
2. Set up six Tor relays behind the OPNsense
3. Wait a few hours when it's heavily loaded.

Expected behavior

The connection should not freeze up, as it didn't on MikroTik using bypass.

Describe alternatives you considered

I tried using Broadcom NICs, increasing the buffer size and disabling logging to disk, but while this delayed the crash it didn't actually fix it.

Screenshots

N/A - I don't see FreeBSD or OPNsense screenshots about this.

Relevant log files

N/A - I don't see FreeBSD or OPNsense logs about this.

Additional context

Bypass using MikroTik is usually reliable if there is no flapping.

I will be moving to NYC next week so I won't have this setup forever.

EDIT: As my family plans to keep the Connecticut home, I'm asking for a tech to move me to XGS-PON. This way I can dump the pfatt bypass.

Environment

• OPNsense 24.7.3 (amd64)
• Minisforum MS-01 (i9-13900H, disabled E-cores, 32GB RAM, 1TB SDD)
• Broadcom BCM5720 for WAN/bypass
• Onboard Intel X710 NIC for LAN
• MonkWho pfatt bypass on a Arris NVG468MQ and an unknown white 4-port Nokia ONT