Open morikplay opened 1 month ago
if 23.10.x works as expected but 24.1 doesn't, the first question is what the difference is in generated ruleset (/tmp/rules.debug
). Would it be possible to collect both on the exact same configuration? Common issues in these cases relate to reply-to
and route-to
rules .
@AdSchellevis Many thanks for the prompt reply!
firmware-->changelog history:
Version Date
24.4.1 (installed) 2024-06-20
24.4 2024-04-30
23.10.3 2024-03-28
Now I notice that 23.x boot environment is no longer visible. I only see the base snapshot manually created my me back in 2022!
bectl list -a
BE/Dataset/Snapshot Active Mountpoint Space Created
default
zroot/ROOT/default NR / 8.44G 2022-10-25 01:28
Perhaps the reboot after 24.x cleared previous boot environments?
What would the sequence of involved steps to furnish you with appropriate /tmp/rules.debug
now be?
bectl create 24.4.1-buggy
opnsense-revert -kr 23.10
A reinstall with 23.10 and update to the latest version in the 23.10 branch would be best indeed, then test if the issue is indeed not there and collect evidence.
If you want to exclude pf
as most likely cause of your issue before reinstalling, I sometimes temporary disable pf
as well using pfctl -d
then check local connectivity and enable it again (pfctl -d
). When local communication works without pf enabled, it's either a policy based routing rule (route-to/reply-to) or a nat issue.
Thank @AdSchellevis. I'll be remote (to FW in question) for the next few weeks. So, a full blown downgrade w/ console access might be a tall order until then.
The following didn't make a dent in the issue:
pfctl -d
root@MorikCage:~ # curl -vi --connect-timeout 5 http://homeassistant.esco.ghaar
* Host homeassistant.esco.ghaar:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.0.58
* Trying 192.168.0.58:80...
* ipv4 connect timeout after 4999ms, move on!
* Failed to connect to homeassistant.esco.ghaar port 80 after 5002 ms: Timeout was reached
* Closing connection
curl: (28) Failed to connect to homeassistant.esco.ghaar port 80 after 5002 ms: Timeout was reached
pfctl -e
I could also attempt a step 2.5 with pfctl -F all
to see whether that helps. But, I doubt it as my rules haven't changed between the two releases.
Would there be any other files besides:
/tmp/rules.debug
dmesg >> /tmp/dmsg.out
from 23.10 which you'd recommend for collection to aid in the diagnosis? I ask so that when I do the downgrade I can collect the right information.well, if it doesn't work when pf is disabled in full (state table doesn't matter in that case), the question is if there is a valid default gateway installed and the interface used has a valid netmask.
netstat -nr4
ifconfig
# netstat -nr4
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 76.214.40.1 UGS igb1
10.10.0.0/24 link#25 U ovpns3
10.10.0.1 link#25 UHS lo0
10.20.30.0/24 link#22 U wg1
10.20.30.1 link#22 UHS lo0
10.20.30.2 link#22 UHS wg1
10.20.30.3 link#22 UHS wg1
10.20.30.4 link#22 UHS wg1
76.214.40.0/22 link#2 U igb1
76.214.40.228 link#2 UHS lo0
127.0.0.1 link#10 UH lo0
192.168.0.0/23 link#21 U vlan0.3
192.168.0.1 link#21 UHS lo0
192.168.2.0/27 link#19 U vlan0.2
192.168.2.1 link#19 UHS lo0
192.168.98.0/24 link#13 U lagg0
192.168.98.1 link#13 UHS lo0
192.168.99.0/24 link#1 U igb0
192.168.99.1 link#1 UHS lo0
192.168.100.0/24 link#15 U vlan0.10
192.168.100.1 link#15 UHS lo0
192.168.120.0/24 link#16 U vlan0.12
192.168.120.1 link#16 UHS lo0
192.168.121.0/24 link#17 U vlan0.12
192.168.121.1 link#17 UHS lo0
192.168.140.0/24 link#18 U vlan0.14
192.168.140.1 link#18 UHS lo0
192.168.250.0/24 link#20 U vlan0.25
192.168.250.1 link#20 UHS lo0
# ping www.google.com
PING www.google.com (142.250.68.68): 56 data bytes
64 bytes from 142.250.68.68: icmp_seq=0 ttl=116 time=5.891 ms
64 bytes from 142.250.68.68: icmp_seq=1 ttl=116 time=7.310 ms
^C
--- www.google.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.891/6.601/7.310/0.709 ms
#ifconfig
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN (lan)
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255
groups: FG_ALL_VLANs FG_CRITICAL_LAN
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Morik_WAN (wan)
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
inet 76.214.40.228 netmask 0xfffffc00 broadcast 76.214.43.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ice0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
media: Ethernet autoselect (25G-AUI <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ice1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
media: Ethernet autoselect (25G-AUI <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ax0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ax1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
enc0: flags=0<> metric 0 mtu 1536
groups: enc
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=20100<PROMISC,PPROMISC> metric 0 mtu 33160
groups: pflog
pfsync0: flags=0<> metric 0 mtu 1500
syncpeer: 0.0.0.0 maxupd: 128 defer: off
syncok: 1
groups: pfsync
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: main_LAGG (opt1)
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
inet 192.168.98.1 netmask 0xffffff00 broadcast 192.168.98.255
laggproto lacp lagghash l2,l3,l4
laggport: ice0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: ice1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg FG_ALL_VLANs FG_CRITICAL_LAN
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=4000000<NOMAP>
groups: vlan
vlan: 1 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: Servers (opt6)
options=4000000<NOMAP>
inet 192.168.100.1 netmask 0xffffff00 broadcast 192.168.100.255
groups: vlan FG_ALL_VLANs FG_CRITICAL_LAN
vlan: 100 vlanproto: 802.1q vlanpcp: 7 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.120: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: Storage (opt3)
options=4000000<NOMAP>
ether f4:90:ea:00:9f:72
inet 192.168.120.1 netmask 0xffffff00 broadcast 192.168.120.255
groups: vlan FG_ALL_VLANs FG_CRITICAL_LAN
vlan: 120 vlanproto: 802.1q vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.121: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: Storage_Backup (opt9)
options=4000000<NOMAP>
inet 192.168.121.1 netmask 0xffffff00 broadcast 192.168.121.255
groups: vlan FG_ALL_VLANs FG_CRITICAL_LAN
vlan: 121 vlanproto: 802.1q vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.140: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: Supervisor (opt4)
options=4000000<NOMAP>
inet 192.168.140.1 netmask 0xffffff00 broadcast 192.168.140.255
groups: vlan FG_ALL_VLANs FG_CRITICAL_LAN
vlan: 140 vlanproto: 802.1q vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: vCamsTraffic (opt2)
options=4000000<NOMAP>
inet 192.168.2.1 netmask 0xffffffe0 broadcast 192.168.2.31
groups: vlan FG_ALL_VLANs
vlan: 2 vlanproto: 802.1q vlanpcp: 1 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.250: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: IoT (opt5)
options=4000000<NOMAP>
inet 192.168.250.1 netmask 0xffffff00 broadcast 192.168.250.255
groups: vlan FG_ALL_VLANs
vlan: 250 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan0.3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: main (opt7)
options=4000000<NOMAP>
inet 192.168.0.1 netmask 0xfffffe00 broadcast 192.168.1.255
groups: vlan FG_ALL_VLANs FG_CRITICAL_LAN
vlan: 3 vlanproto: 802.1q vlanpcp: 2 parent interface: lagg0
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wg1: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
description: i_wireguard (opt10)
options=80000<LINKSTATE>
inet 10.20.30.1 netmask 0xffffff00
groups: wg wireguard
nd6 options=9<PERFORMNUD,IFDISABLED>
ovpns1: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
groups: tun openvpn
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ovpns2: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
groups: tun openvpn
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ovpns3: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.10.0.1 netmask 0xffffff00 broadcast 10.10.0.255
groups: tun openvpn
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 65678
Pictorial view also attached for easier readability.
Just to be sure, internet hosts also don't work or only access to 192.168.0.58
?
You may remove sensitive data from the output by the way, for debugging we don't need mac addresses and exact netblocks.
@AdSchellevis, thank you. I'll remove MAC addresses and such shortly.
Yes, access to internet (NAT) isn't a problem. The issue is only for traffic which originates from OpnSense e.g. syslog to a syslog server on lan, or Telegraf metrics destined for influxdb on lan, CrowdSec http notification to a LAPI on lan, simple curl requests to anywhere on any lan etc.
I wouldn't suspect the firewall to be honest, my next step would be to capture the traffic on both ends (packet capture on the firewall and on the target for the traffic between both hosts).
These type of issues usually relate to wrong gateways on the client in which case traffic does not return to the expected host.
Thank you for your continued guidance @AdSchellevis . Initially, I too had the suspicion that it must have been me (by virtue of network configuration changes) which would've caused this issue. But, few aspects don't support that conclusion:
tcpdump
captures at Opnsense interface (opt6) from initial message, one can see that opt6 (same goes for other opt interfaces) does receive a SYN-ACK in response to its SYN. But, application logic (i.e. something above the TCP/IPv4 stack in opnsense) does not receive delivery of the SYN-ACKs. This causes TCP/IPv4 stack on opnsense to repeat SYN. Re-pasting for brevity
Servers
vlan0.100 2024-06-28
07:37:50.442037 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x8070 (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292126707 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:50.442400 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xe967 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838080763 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:51.442697 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x7c87 (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292127708 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:51.443231 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xe57e (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838081764 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:52.462713 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xe182 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838082784 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:53.642675 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x73ef (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292129908 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:53.643161 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xdce6 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838083964 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:55.662758 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xd502 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838085984 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:37:57.842474 f4:90:ea:00:9f:72 00:50:56:82:d8:b4 ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.1.31315 > 192.168.100.21.8080: Flags [S], cksum 0x6387 (correct), seq 445912424, win 65535, options [mss 8960,nop,wscale 12,sackOK,TS val 1292134108 ecr 0], length 0
Servers
vlan0.100 2024-06-28
07:37:57.842885 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xcc7e (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838088164 ecr 1292126707,nop,wscale 9], length 0
Servers
vlan0.100 2024-06-28
07:38:01.966765 00:50:56:82:d8:b4 f4:90:ea:00:9f:72 ethertype IPv4 (0x0800), length 74: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
192.168.100.21.8080 > 192.168.100.1.31315: Flags [S.], cksum 0xbc62 (correct), seq 3873949677, ack 445912425, win 43440, options [mss 1460,sackOK,TS val 3838092288 ecr 1292126707,nop,wscale 9], length 0
curl
and nc
exhibit really odd behavior. Latter recognizes tcp/443 as udp/443. #nc -4znvw 10 192.168.0.58 443
Connection to 192.168.0.58 443 port [tcp/*] succeeded!
<!-- immediately following which another series of requests fail -->
...
#nc -4znvw 10 192.168.0.58 443
nc: connect to 192.168.0.58 port 443 (tcp) failed: Operation timed out
# nc -4znvw 10 192.168.0.58 443
nc: connect to 192.168.0.58 port 443 (tcp) failed: Operation timed out
tcpdump
from the peer (192.168.100.21:8080). I picked a different host (than .100.6
) in VLAN-100 on a non-standard port to illustrate the point further. As can be seen, the host keeps getting re-transmissions of SYN from FW (caused as a result of curl -vi --connect-timeout 5 http://192.168.100.21:8080
. At the FW interface, NIC receives the packet, but doesn't respond with ACK.
from_host.zipI'm a bit out of clues I'm afraid, I suspect there is a logical explanation, but tracking this exceeds my current community support time.
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
Describe the bug
I posted the issue here but have yet to receive a response. Hence, this report. Please accept my apologies for potentially increasing the workload of opnsense's awesome team!
Upgraded from 23.x (latest release for business) to 24.4.1. Still on ISC DHCP with only IPv4; KEA not yet enabled. Intra- and Inter- VLAN packet routing works like a charm. No configuration or rule changes were made since. However, any outbound TCP (or sometimes UDP) packets in FW-->(any)VLAN direction on standard (e.g. 80, 443) or non-standard (e.g. 8080) ports do not get processed. ICMP pings to the same LAN hosts are fine. All other intra-VLAN nodes, inter-VLAN nodes, and even external nodes (e.g. web client on a cellular connection) can reach the address:ports in question. Just not the packets originating from FW. Example of such packets are telegraf metrics, crowdsec-lapi requests, or even simple requests from ssh/console like
curl -vi --connect-timeout n <url>
timeout.Reverting to previous environment restores normalcy.
Steps to demonstrate issue
ping 192.168.100.6 PING 192.168.100.6 (192.168.100.6): 56 data bytes 64 bytes from 192.168.100.6: icmp_seq=0 ttl=64 time=0.369 ms ..
traceroute 192.168.0.58 traceroute to 192.168.0.58 (192.168.0.58), 64 hops max, 40 byte packets 1 homeassistant (192.168.0.58) 0.464 ms 0.167 ms 0.175 ms ... traceroute 192.168.100.6 traceroute to 192.168.100.6 (192.168.100.6), 64 hops max, 40 byte packets 1 portainer (192.168.100.6) 0.359 ms 0.133 ms 0.241 ms
@portainer:~$ nc -4nzvw 5 192.168.0.58 443 Connection to 192.168.0.58 443 port [tcp/*] succeeded! @portainer:~$ nc -4nuzvw 5 192.168.0.58 443 (base) maumau@portainer:~$
@portainer:~$ curl -ki --connect-timeout 5 https://homeassistant.esco.ghaar HTTP/2 200 server: nginx date: Sat, 06 Jul 2024 18:48:24 GMT content-type: text/html; charset=utf-8 content-length: 4148 referrer-policy: no-referrer x-content-type-options: nosniff x-frame-options: SAMEORIGIN strict-transport-security: max-age=31536000; includeSubDomains ... (truncated the full HTTP response)
nc -4nzvw 5 192.168.0.58 443 nc: connect to 192.168.0.58 port 443 (tcp) failed: Operation timed out nc -4nuzvw 5 192.168.0.58 443 Connection to 192.168.0.58 443 port [udp/*] succeeded! ...
...
curl -kvi --connect-timeout 5 https://homeassistant.esco.ghaar
Expected behavior
Opnsense should establish HTTPS/TCP connection, or even basic TCP connection to hosts on various VLANs.
Describe alternatives you considered
Please refer to above. Reverting to Opnsense 23.x (last known business version) restores normalcy.
Screenshots
If applicable, add screenshots to help explain your problem.
Relevant log files
If applicable, information from log files supporting your claim.
Additional context
Few additional considerations which i grappled with:
Does this issue always happen? Mostly yes, but every so often (once in a few hours or so) a TCP request in FW-->Host direction slips through. Evidence:
What is going on at TCP/IPv4 level? a. Are the FW originating HTTP(s) over TCP packets sent over the wire? Yes b. If so, is the switch network eating it up? (due to say bad VLAN configuration No c. Is the receiving host not responding at TCP level? No. Receiving host does issue TCP SYN ACKs d. Are receiving hosts packets blocked by FW rules? Answer: No e. Are receiving hosts packets received at the FW interface? Answer: Yes
Packet capture from opt6 (VLAN 100) to illustrate packet behavior can be seen below. Same behavior for main/opt7 interface is also observed. :
mtu:9000
(downsampled at ether layer) with hostmtu:1500
(downsampled at ether layer). Changing host mtu to be same as FW results in same behavior. Please see attachedtcpdump.zip
which includespcap
andjson
.tcpdump.zip
ice
card involved? Doesn't seem so, Also, if that were to be the case then it should manifest across all VLANs? There are certain oddities about ice messages but these seemed also to be there in 23.x (if memory serves me right).Environment
Version 24.4.1
Architecture amd64
Commit 77b950d6f
Mirror https://opnsense-update.deciso.com/${SUBSCRIPTION}/FreeBSD:13:amd64/24.4 CPU AMD EPYC 3251 8-Core Processor (8 cores, 8 threads) HW DEC4040