PfSense 23.01/2.7 (FreeBSD 14.0)

[bigjohns97]

With the new release coming up and quite a few changes I figured it was time to start a discussion on bypass scripts for the new version of PfSense about to release.

A couple of notes that hopefully saves some people some time.

https://redmine.pfsense.org/issues/12821?next_issue_id=12820

Looks like we won't have to be using netgraph moving forward which would be nice

With the introduction of native PCP VLAN0 tagging in pfSense Plus 23.01 and the new bridge filtering to pass along EAP traffic from another interface natively integrated into pfSense Plus

Which is good because the src isn't available for 14.0 yet so we wouldn't be able to compile a new Intel driver against it.

But this does mean we are going to need to build a new script with new commands to tag VLAN and authenticate EAP traffic.

[bigjohns97]

Some extra threads for reference

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224961 https://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+14.0-CURRENT&arch=default&format=html

I tried just about everything to get this thing to listen, you do have to remove the previously compiled drivers from the other thread or the system halts, also you can't run any pfatt scripts because the ngeth0 interface comes up with a random MAC, and either way it doesn't seem like it would work based upon the above threads.

[bigjohns97]

So the good news is I was able to get it working with the old school script but it's still using netgraph. :(

Maybe someone with a little more skills and time can get it working without netgraph, I've used up all of my political capital with the family today :)

I will post the script below (see attached)

As you can see I only added the "/sbin/ifconfig $ONT_IF -vlanhwfilter" under the enabling promisc section for my supplicant option, if you use the bridge go ahead and do the same under the bridge section of the script. (should only be required for Intel NIC's)

This doesn't require any custom driver anymore so at least there is a small win there. 2301_bypass.txt

[dmitripr]

If I'm reading the comment correctly, I think someone solved this without using the script: https://redmine.pfsense.org/issues/12070#note-7

If someone can contact them and ask about the procedure?

[bigjohns97]

If I'm reading the comment correctly, I think someone solved this without using the script: https://redmine.pfsense.org/issues/12070#note-7

If someone can contact them and ask about the procedure?

I just tried what they suggested tonight and wasn't able to get it working, I posted in that thread in hopes that he will respond.

[Shadow00Caster]

Is there any known fix for this?

[liuhaotian]

While trying out 23.01 RC, I found another way to get ATT working by using same VPID for WAN and OPT and keep pfSense and ATT RG as dual gateway, confirmed on 22.05 without pfatt. hope if helps: https://navigateheadwinds.blogspot.com/2023/02/pfsense-att-dual-gateway.html

[bigjohns97]

Is there any known fix for this?

Known fix for what?

I posted the edit of the original script to use with Intel NIC.

Still working on trying to figure out how to not require the full script and take ngeth out of the equation.

[TTK-199837]

Sorry, but could someone clarify for me? If I have pfatt working fine on 22.05 will upgrading to 23.01 break it?

[bigjohns97]

It depends are you using the custom compiled driver?

[TTK-199837]

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

[bigjohns97]

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.

23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.

This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.

GL

[TTK-199837]

Thank you for that explanation. I do remember doing something with the intel NICs on my SG5100 in one of the recent updates.

On Sun, Feb 19, 2023, at 6:29 AM, bigjohns97 wrote:

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.

23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.

This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.

GL

— Reply to this email directly, view it on GitHub https://github.com/MonkWho/pfatt/issues/79#issuecomment-1436004131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AND3BCWPUQK3OR6IQ3LFNR3WYIU47ANCNFSM6AAAAAAT3LWZAM. You are receiving this because you commented.Message ID: @.***>

[unawarez]

@bigjohns97 are you using wpa-supplicant/is that relevant in this issue? I've been trying to navigate this whole thing across different threads/bugs and I'm thinking maybe there's a mixture of feedback between wpa-supplicant and passthrough setups out there. FWIW I have a supplicant setup and also would like to get rid of netgraph. Even with netgraph I've always needed wpa-supplicant running on the "raw" interface so that its traffic doesn't get VLAN tags. I've been trying to come up with some clever setup with epair and a bridge or something like that but can never get the return traffic routed to the right interface.

[bigjohns97]

@bigjohns97 are you using wpa-supplicant/is that relevant in this issue? I've been trying to navigate this whole thing across different threads/bugs and I'm thinking maybe there's a mixture of feedback between wpa-supplicant and passthrough setups out there. FWIW I have a supplicant setup and also would like to get rid of netgraph. Even with netgraph I've always needed wpa-supplicant running on the "raw" interface so that its traffic doesn't get VLAN tags. I've been trying to come up with some clever setup with epair and a bridge or something like that but can never get the return traffic routed to the right interface.

Yes I am using supplicant.

However, in my testing with this I never did the wpa_supplicant before applying the PCP values, I can try this later on today and see if it makes a difference.

[unawarez]

Well I'm posting this from a zero-netgraph supplicant setup after a reboot 😎

~And as a side-effect my bufferbloat score has gone from B to A, without any traffic shaping (which I couldn't set up before anyway due to using netgraph)~ I think my B was from pfSense 22.05, can't remember if I tried on 23.01 before this

My approach was using wpa_cli's -a flag which lets you run a hook script every time the interface dis/connects. I use that to run ifconfig $ifname pcp 0 every time wpa_supplicant connects. I have no idea how well it handles disconnects though. I'll attach my current minimal scripts. I'm running this as a shellcmd at startup.

Edit: the wpa_cli hook does not handle disconnects well. When my fiber dropped out I had to reboot the switch to get IPv6 back. The disconnect hook never fired. That really makes me want to use dhcpcd hooks but I don't have time right now to figure all that out.

pfatt_sup14.txt pfatt_sup14_action.txt

[bigjohns97]

Well I'm posting this from a zero-netgraph supplicant setup after a reboot 😎

~And as a side-effect my bufferbloat score has gone from B to A, without any traffic shaping (which I couldn't set up before anyway due to using netgraph)~ I think my B was from pfSense 22.05, can't remember if I tried on 23.01 before this

My approach was using wpa_cli's -a flag which lets you run a hook script every time the interface dis/connects. I use that to run ifconfig $ifname pcp 0 every time wpa_supplicant connects. I have no idea how well it handles disconnects though. I'll attach my current minimal scripts. I'm running this as a shellcmd at startup.

pfatt_sup14.txt pfatt_sup14_action.txt

I wasn't able to get mine to authenticate using these scripts even tried -vlanhwfilter as an extra step to see if it was that.

No-Go

[unawarez]

I wasn't able to get mine to authenticate using these scripts even tried -vlanhwfilter as an extra step to see if it was that.

Ugh sorry. FWIW I'm using a Netgate SG-3100 which is 32-bit ARM and Marvell network interfaces. I can provide command or log output of anything that would help

[unawarez]

On reboot number 2, wpa_supplicant logged a failed auth, then tried again 30 seconds later and succeeded

[bigjohns97]

Are you running this as a earlyshellcmd?

That is how I was trying to use it and once booted up I could run ifconfig and noticed that I didn't have pcp assigned to the interface.

I am wondering if you could post how you came up with the order of things.

Judging by your script.

1. bring up interface and set promisc mode
2. remove any previous pcp attribute
3. run wpa auth along with connected watcher (4,5 below)
4. if not connected add pcp attribute
5. once connected remove pcp attribute

The whole time wpa and connected watcher stays running in the background.

Is this accurate?

[bigjohns97]

BTW I noticed you reference en interface so your WAN is intel en?

[bigjohns97]

Tried again this morning, toggling everything I could

with and without pcp with and without vlanhwfilter using my RG MAC as the wpa identity using the purchased certs wpa identity

I could never get the wpa process to successfully authenticate, and I waited until it told me it failed.

I am wondering if this has anything to do with my certs being associated to another MAC than my RG, I still assign my RG MAC to my ngeth and my igb0 interface regardless but it just seems to work when using ngeth.

At a dead end, up for trying anything at this point if anyone has any ideas.

[unawarez]

BTW I noticed you reference en interface so your WAN is intel en?

No sorry that was pretty bad on my part, en123 is a placeholder name, and it's a Linuxism at that, where every wired iface gets named en<something>. My real WAN is mvneta2

Are you running this as a earlyshellcmd?

Only tried as a shellcmd so far. I want to try as an earlyshellcmd soon.

I am wondering if you could post how you came up with the order of things.

Judging by your script.

1. bring up interface and set promisc mode

2. remove any previous pcp attribute

3. run wpa auth along with connected watcher (4,5 below)

4. if not connected add pcp attribute

5. once connected remove pcp attribute

The whole time wpa and connected watcher stays running in the background.

Is this accurate?

Yep that's all accurate. wpa_cli is a second long-running process alongside wpa_supplicant. I was looking for any of the standard daemons that might support external hooks. And actually I think long-term doing this as dhcpcd hooks instead of supplicant hooks might be a little more reliable, but that's just a hunch, and idek if you can add config options like that to the dhcpcd instances that pfSense automatically runs for interfaces.

To get here I'd been experimenting with setting my WAN interface in pfSense to the real WAN iface (mvneta2 in my case) instead of ngeth0. Through a combo of watching the system logs in the web UI and doing some packet captures I saw for whatever reasons in my setup, my ONT will only respond to 802.1X frames that do NOT have any 802.1Q tags, and will only respond to regular traffic that DOES have 802.1Q tags. I toggled the tags by running ifconfig mvneta2 pcp 0 or -pcp and occasionally dis/enabling the interface. I was struggling for a long time to get a non-netgraph equivalent of two network interfaces that had the same external MAC, so I could set pcp 0 on one and leave wpa_supplicant on the other. Using connection hooks to toggle it instead on one interface was a shower thought and only worked because these daemons happen to support hooks.

I don't know if promisc mode is still needed. That's left from original pfatt.

For debugging this script live I was ssh'd in and alternating between running this script and running a pfatt_cleanup.txt script I threw together. All the while watching the system logs in web UI for output from supplicant/etc.

I am wondering if this has anything to do with my certs being associated to another MAC than my RG, I still assign my RG MAC to my ngeth and my igb0 interface regardless but it just seems to work when using ngeth.

My RG MAC is forced on mvneta2 in the web UI. Can't help with the different cert identity unfortunately.

If you haven't already I'd recommend getting a packet capture in the web UI of a successful supplicant handshake with your known good setup, download it and check it out in Wireshark or something. This probably means booting normally, physically disconnecting the cable, starting capture, and then reconnecting the cable. That should at least give a baseline of if it's expecting .Q tags or not.

[bigjohns97]

Well unfortunately the packet captures didn't produce any valuable information.

The EAPOL start frame looks exactly the same on your script and the original pfatt script.

The only difference is the response which through the pfatt script is a success and your script is replies on VLAN 0 requesting an identity.

I'm going to have to wait for someone with more skills than myself to come up with a solution.

[unawarez]

Your traffic sounds similar to other reports I’ve seen in different threads. Maybe there’s more variance in ONT hardware (or firmware?) than folks have been thinking :( I tried getting my ONT’s model number but it’s conspicuously unlabeled.

I’m not sure how I’d approach this then for one and only one untagged packet. I don’t know anything about freebsd pf but if it’s able to match on the start packet and only the start packet, I bet there’d be a way from there to redirect or modify that start packet, and just have your WAN interface using pcp 0 all the time.

On Sun, Feb 26, 2023, at 15:13, bigjohns97 wrote:

Well unfortunately the packet captures didn't produce any valuable information.

The EAPOL start frame looks exactly the same on your script and the original pfatt script.

The only difference is the response which through the pfatt script is a success and your script is replies on VLAN 0 requesting an identity.

— Reply to this email directly, view it on GitHub https://github.com/MonkWho/pfatt/issues/79#issuecomment-1445469206, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3HIGZBNH7WLW5BCEVFSWCDWZPBRBANCNFSM6AAAAAAT3LWZAM. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[bigjohns97]

When I was running ESXi I would just create the NIC with my RG MAC at the virtual layer and tag VLAN 0 at the virtual layer as well and everything would work fine.

When you do an ifconfig, do you see an ether address as well as a hwaddr? Are they the same or different? Are they your RG MAC?

[unawarez]

The MACs are different. ether is my RG MAC, but hwaddr is 00:08:a2:something (same prefix for all my real interfaces)

[bigjohns97]

The ngeth interface had a different hwaddr as well. I guess I need to dig into that and really see what makes that different than what I am doing.

[unawarez]

When I was running ESXi I would just create the NIC with my RG MAC at the virtual layer and tag VLAN 0 at the virtual layer as well and everything would work fine.

I find this especially weird...is there some standards-edge-case about eapol starts never supposed to be tagged? Otherwise if ESXi was tagging the start packet (which I would've assumed, just out of a hope for sanity) then why was the ONT accepting those starts on VLAN 0 but not accepting on pfSense if you just put supplicant on an interface with pcp 0 all the time?

The ngeth interface had a different hwaddr as well. I guess I need to dig into that and really see what makes that different than what I am doing.

Let me paste my supplicant conf too. I do have a cert that matches my provisioned RG MAC, so that is different, but I can't imagine that affecting this .Q stuff.

eapol_version=2
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/pfatt/wpa/ca.pem"
        client_cert="/conf/pfatt/wpa/client.pem"
        eap=TLS
        eapol_flags=0
        identity="[redacted RG MAC]"
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/pfatt/wpa/private.pem"
}

I guess you could also try adding a line into the script that forces the RG MAC on the interface before it comes up. I'm relying on pfSense config to do that, and I could imagine that working for shellcmd but not earlyshellcmd, if the script happens before pfSense applies all the user config options.

[bigjohns97]

I tried the line to the script setting the ether addr but one thing I just noticed is your eapol_version is 2 and mine is 1.

Checking the original pfatt script it looks like it has it set to 2 as well, will check to see if this makes a difference a little later on tonight.

[bigjohns97]

Still a negative, even trued using my identity MAC as my spoofed MAC on my WAN interface this time thinking maybe that would work.

no go

Also tried setting a pcp of 7 since that is what the ONT was replying with when asking for my identity, still didn't work.

Giving up, surely someone else will come along and test out your script and see if it's me or maybe you just have something special with those realtek interfaces.

[bigjohns97]

@unawarez would you mind posting your interface options?

Here are mine RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP for a while we had to disable vlanhwfilter for the pfatt script to work whenever they moved to a consolidated igb and em intel driver. (I still do this BTW and have been trying this in my testing, just wondering if there are more I should be disabling)

I am wondering if one of these are disabled on your driver by default and may explain our different experiences.

[unawarez]

Sure:

mvneta2: flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
    description: WAN
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>

so you have these I don't: TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP

and I have LINKSTATE

edit: I just discovered ifconfig -m $IFNAME which also displays the available capabilities. The only extra one I have that is disabled is LRO. I notice there's no VLAN_HWFILTER even as an option for mine...

capabilities=804bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,LINKSTATE>

[mdelise]

Hello, looking for some help. I'm currently using pfsense 2.4.5 with pfatt in supplicant with working certs.

I purchased a DFP-34X-2C2 ONT stick, cloned the white ATT GPON ONT and installed in a pcie card in the pfsense server.

I'm trying to figure out what necessary changes to the pfatt script are required? Netgraph is still needed to handle the VLAN0 situation, but I also need the WAN VLAN Tagged on 962 because of where internet is available from the DFP ONT.

# omcicli mib get 84
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VlanTagFilterData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=================================
EntityID: 0x1102
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x02
NumOfEntries: 1
=================================
=================================
EntityID: 0x110b
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x10
NumOfEntries: 1
=================================

[aholmes55]

I have both supplicant and bridge/tethered bypass working with netgraph on pfSense 23.01 on Netgate hardware (Intel igb nics) with ipv6. I have a BGW210 gateway. The included Intel drivers work fine for this release but you do have to add the vlanhwfilter flags to ifconfig in both bridge and supplicant modes per Issue 65. I am using certificates from an old device with a different MAC than my assigned gateway. The only issue I've seen is pfSense struggles to update packages (super slow or fails) when ipv6 is enabled. Without ipv6 it is quick.

pfatt.txt

[bigjohns97]

Hello, looking for some help. I'm currently using pfsense 2.4.5 with pfatt in supplicant with working certs.

I purchased a DFP-34X-2C2 ONT stick, cloned the white ATT GPON ONT and installed in a pcie card in the pfsense server.

I'm trying to figure out what necessary changes to the pfatt script are required? Netgraph is still needed to handle the VLAN0 situation, but I also need the WAN VLAN Tagged on 962 because of where internet is available from the DFP ONT.

# omcicli mib get 84
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VlanTagFilterData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=================================
EntityID: 0x1102
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x02
NumOfEntries: 1
=================================
=================================
EntityID: 0x110b
FilterTbl[0]: PRI 0,CFI 0, VID 962
FwdOp:  0x10
NumOfEntries: 1
=================================

Please open up another thread to discuss this

[TTK-199837]

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3.

23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine.

This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best.

GL

Hi again! Requesting some more hand holding if you do not mind ;)

You mention "just disable hwvlanfilter", however I am not familiar with this setting, or having ever enabled it. After some exploring, I believe these steps should remove / disable the old custom intel drivers, and have a working router, please let me know if I am correct:

• Reverse steps found here: https://github.com/MonkWho/pfatt/issues/67#issuecomment-1043358822

• Remove old intel driver if_igb.ko from /boot/kernel

• Remove if_igb_load="YES" and if_igb_name="/boot/modules/if_igb.ko" from /boot/loader.conf

• (?) should I keep my current pfatt.sh unchanged? It seems a LOT different than the couple posted here, I believe because I am using bridge mode? Do I, or should I modify it in anyway? (https://github.com/TTK-199837/pfatt/blob/master/bin/pfatt.sh)

• Update via GUI, and we are up and running?

Thanks for your time!

[gpz1100]

@bigjohns97 Did you ever get the igb nonsense figured out?

I have a i211 passed through using proxmox. Get absolutely no eapol traffic on tcpdump in 23.01. I can have proxmox handle the vlan0 tagging which results in virtio drivers for wan. This works, but at line speeds, cpu usage is way high (top in proxmox).

In sophos utm, with the i211 passsed, i'm seeing roughly 1/4 of the cpu usage doing line speed upload/download tests. Would like to move to pf, but this is not acceptable.

[bigjohns97]

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3. 23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine. This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best. GL

Hi again! Requesting some more hand holding if you do not mind ;)

You mention "just disable hwvlanfilter", however I am not familiar with this setting, or having ever enabled it. After some exploring, I believe these steps should remove / disable the old custom intel drivers, and have a working router, please let me know if I am correct:

• Reverse steps found here: Intel igb/em Interfaces Broken on 2.6/22.01+ #67 (comment)
• Remove old intel driver if_igb.ko from /boot/kernel
• Remove if_igb_load="YES" and if_igb_name="/boot/modules/if_igb.ko" from /boot/loader.conf
• (?) should I keep my current pfatt.sh unchanged? It seems a LOT different than the couple posted here, I believe because I am using bridge mode? Do I, or should I modify it in anyway? (https://github.com/TTK-199837/pfatt/blob/master/bin/pfatt.sh)
• Update via GUI, and we are up and running?

Thanks for your time!

You should check out the script I posted at the very top of this thread (3rd post) https://github.com/MonkWho/pfatt/files/10418690/2301_bypass.txt

I made this edit for supplicant mode (bolded part is my edit) /usr/bin/logger -st "pfatt" "enabling promisc for $ONT_IF..." /sbin/ifconfig $ONT_IF up /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter

To do the same for bridge mode just add this part to bridge section /usr/bin/logger -st "pfatt" "enabling promiscuous mode..." /sbin/ifconfig $EAP_BRIDGE_IF promisc /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter

BTW you are correct on how to remove all of the previous custom driver stuff from the previous version.

Also if you still want to use your script add something like this to the end

echo -n "$(getTimestamp) enabling promiscuous mode on $ONT_IF... " /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter echo "OK!"

Or you can do a whole new section if you want

echo -n "$(getTimestamp) disabling vlanhwfilter on $ONT_IF... " /sbin/ifconfig $ONT_IF -vlanhwfilter echo "OK!"

Hope this helps

[bigjohns97]

@bigjohns97 Did you ever get the igb nonsense figured out?

I have a i211 passed through using proxmox. Get absolutely no eapol traffic on tcpdump in 23.01. I can have proxmox handle the vlan0 tagging which results in virtio drivers for wan. This works, but at line speeds, cpu usage is way high (top in proxmox).

In sophos utm, with the i211 passsed, i'm seeing roughly 1/4 of the cpu usage doing line speed upload/download tests. Would like to move to pf, but this is not acceptable.

If you want try disabling hardware checksum offload in pfsense on the guest and let me know if that solves your CPU usage issues.

(all three of these should be checked)

As to me getting where I would like to be I still haven't been able to get a good understanding why we can't remove netgraph and get it to work, I did get this message from ccope

Which would be great if he shared but I didn't want to push anymore than I already have.

[gpz1100]

@bigjohns97 Unchecking all 3 of those leads to a slight marginal improvement in cpu usage for virtio. How ever the webgui becomes painfully sluggish.

By default the bottom 2 options are check marked.

Can you clarify what is the state of eapol traffic with igb (i211) driver/nic? Should this be working somehow?

[TTK-199837]

How can I check for that? I remember it breaking the last two times I updated, so I am being very caution this time. Don't need the household yelling at me for another multi hour downtime haha. Thanks for the help!

If you have an Intel igb or em based NIC and had to load the custom driver last update then you will have an issue as those were compiled with the previous version of PfSense which was running on an older version of FreeBSD 12.3. 23.01 is running on FreeBSD 14 so you can't use that driver, you can however just disable hwvlanfilter and use the native driver and pfatt script (feel free to see mine above and compare to yours to see what I changed) and then everything should work fine. This is probably the biggest change in a long time with them moving two major revisions of FreeBSD ahead and without me knowing your setup I can only speak to pfatt and nothing else so make sure you do your research and prepare for the worst and hope for the best. GL

Hi again! Requesting some more hand holding if you do not mind ;) You mention "just disable hwvlanfilter", however I am not familiar with this setting, or having ever enabled it. After some exploring, I believe these steps should remove / disable the old custom intel drivers, and have a working router, please let me know if I am correct:

• Reverse steps found here: Intel igb/em Interfaces Broken on 2.6/22.01+ #67 (comment)
• Remove old intel driver if_igb.ko from /boot/kernel
• Remove if_igb_load="YES" and if_igb_name="/boot/modules/if_igb.ko" from /boot/loader.conf
• (?) should I keep my current pfatt.sh unchanged? It seems a LOT different than the couple posted here, I believe because I am using bridge mode? Do I, or should I modify it in anyway? (https://github.com/TTK-199837/pfatt/blob/master/bin/pfatt.sh)
• Update via GUI, and we are up and running?

Thanks for your time!

You should check out the script I posted at the very top of this thread (3rd post) https://github.com/MonkWho/pfatt/files/10418690/2301_bypass.txt

I made this edit for supplicant mode (bolded part is my edit) /usr/bin/logger -st "pfatt" "enabling promisc for $ONT_IF..." /sbin/ifconfig $ONT_IF up /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter

To do the same for bridge mode just add this part to bridge section /usr/bin/logger -st "pfatt" "enabling promiscuous mode..." /sbin/ifconfig $EAP_BRIDGE_IF promisc /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter

BTW you are correct on how to remove all of the previous custom driver stuff from the previous version.

Also if you still want to use your script add something like this to the end

echo -n "$(getTimestamp) enabling promiscuous mode on $ONT_IF... " /sbin/ifconfig $ONT_IF promisc /sbin/ifconfig $ONT_IF -vlanhwfilter echo "OK!"

Or you can do a whole new section if you want

echo -n "$(getTimestamp) disabling vlanhwfilter on $ONT_IF... " /sbin/ifconfig $ONT_IF -vlanhwfilter echo "OK!"

Hope this helps

You are a lifesaver. Thank you very much! Update went smoothly and as planned!

[gpz1100]

Wanted to update. Yes, the netgraph method still works in 23.01 with those ifconfig switches.

What's unclear; Is there still a need for netgraph?

With a i211, I could not get it to process any eapol traffic without netgraph.

[aholmes55]

Want to point out that @bigjohns97's script "2301_bypass.txt" doesn't have the -vlanhwfilter on the bridge mode only on the supplicant mode. Both modes need the -vlanhwfilter on the ifconfig $ONT_IF promisc command.

[dreamdenizen]

@bigjohns97 thanks for posting how you got it working and a copy of your script. Got supplicant working for the first time after 1 year on bridged, and now I'm on 23.01 to boot. Thanks again!

[ChronicledMonocle]

FYI to those on this thread, all of the UI elements are added into pfSense Plus 23.05 to do this without a script and no netgraph required. You can enable promiscuous mode for the modem interface, set a PCP value for the WAN interface, and setup the 802.1X bridging using the new Ethernet Rules UI under Firewall --> Rules (Need to go to System --> Advanced --> Firewall & NAT --> Ethernet Filtering and enable the option first). I'm currently running an ATT connection bypassed with the bridge method (not wpa_supplicant) and it's been running for a few weeks trouble free.

If you are running an e1000-based NIC on either/both of your modem and WAN interfaces, you will need to disable vlanfwfilter still with an earlyshellcmd script (ifconfig igb# -vlanhwfilter) but any other NIC should just work.

Screenshot of the Ethernet Rules in Action:

[dmitripr]

@ChronicledMonocle Great news! Thanks for sharing. Is there a link to step-by-step instructions on setting this up via UI?

[dmitripr]

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.

• Upgrade to 23.05-RC to get the option for Ethernet Filtering
• Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)
• Added the new USB interface via Interfaces -> Assignments
• The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None
• I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem
• Added the two rules mentioned by @ChronicledMonocle in the screenshot under Firewall -> Rules -> Ethernet
• Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X

This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I tested a restart of the pfSense box and the modem, and this setup survived the restart.

[ChronicledMonocle]

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.

* Upgrade to 23.05-RC to get the option for Ethernet Filtering

* Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)

* Added the new USB interface via Interfaces -> Assignments

* The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None

* I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem

* Added the two rules mentioned by @ChronicledMonocle  in the screenshot under Firewall -> Rules -> Ethernet

* Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X

This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I haven't tried to see if this will survive a reboot of the modem and the pfSense router yet.

We have a documentation recipe going up on docs.netgate.com that will have a configuration example of how to do this step by step. I wrote this with input from the pfSense Plus TAC and Development teams for editing and feedback. Attached is a PDF of the draft before it goes live there. Bear in mind you need to be on pfSense Plus 23.05 RC for these instructions to work: Configuring WAN Connectivity for 802.1X Authentication Bridging and VLAN0 PCP Tagging.pdf

[ChronicledMonocle]

Also, the e1000 bug with vlanhwfilter should be fixed in the latest 23.05 RC release, so should be no need to disable vlanhwfilter anymore on those NICs.

[dmitripr]

After playing around with settings I got it to work. A few notes/steps below may help others. Referring back to what @ChronicledMonocle mentioned a couple of posts above.

* Upgrade to 23.05-RC to get the option for Ethernet Filtering

* Attached a USB NIC for my connection to the modem (if you have a free port on your pfSense you can use that)

* Added the new USB interface via Interfaces -> Assignments

* The only setting I changed on the new USB NIC interface for the Modem is to enable "Promiscuous Mode"; left the IPv4 and IPv6 Configuration types as None

* I couldn't find the "PCP" setting on the WAN interface (I guess it's called Priority Tag?), I left it blank; just ensured that the MAC Address field is set to the MAC of the AT&T Modem

* Added the two rules mentioned by @ChronicledMonocle  in the screenshot under Firewall -> Rules -> Ethernet

* Need to use the Advanced Options and set the Bridge To field; and the first rule needs to have the Protocol set to IEEE 802.1X

This worked for me without any restarts (except when upgrading pfSense to 23.05-RC). I haven't tried to see if this will survive a reboot of the modem and the pfSense router yet.

We have a documentation recipe going up on docs.netgate.com that will have a configuration example of how to do this step by step. I wrote this with input from the pfSense Plus TAC and Development teams for editing and feedback. Attached is a PDF of the draft before it goes live there. Bear in mind you need to be on pfSense Plus 23.05 RC for these instructions to work: Configuring WAN Connectivity for 802.1X Authentication Bridging and VLAN0 PCP Tagging.pdf

Good stuff. I would also recommend adding DHCPV6 Server and RA basic/recommended settings to the document.