Open ChronicledMonocle opened 2 years ago
Can confirm its not working after an upgrade. Following the troubleshooting instructions shows that the modules have loaded. PFatt logs dont show anything.
Can confirm it is broken for me as well, running supplicant
Same issue. Not grabbing DHCP.
Edit: I am using the WPA supplicant method.
Want to confirm that reverting to 2.5.2 or 21.05.2 immediately restores internet for me after setting everything back up.
Want to confirm that reverting to 2.5.2 or 21.05.2 immediately restores internet for me after setting everything back up.
Yes, It was an absolute pain in the a**, but restoring to 21.05.2 immediately fixed it for me. IPv6 wouldn't grab for an hour or so but finally started working.
Also, I posted on the Netgate Forums. If anyone else wants to add anything over there here is the link. https://forum.netgate.com/topic/169882/22-01-2-6-0-upgrade-broke-dhcp-on-wan-interface-with-custom-startup-script
I am having the same problem and now my WireGuard and other tools don't work and can't get them to work.
Yep - supplicant not working for me either. The last time a new version of pfsense broke pfatt Matt Johnson submitted this issue to pfsense redmine. Should we do that here? Here is the issue that originated the whole thing.
It also broke mine after update. Per the docs, I ran "tcpdump -ei ONT_IF" and "tcpdump -ei RG_IF", which should filter and capture link layer information (2), on my interfaces and captured 0 packets from RG_IP and only the bridged DHCP traffic on the ONT_IF interface.
I reset netgraph, which removes the hooks, rebooted the gateway and modem with tcpdump running and captured 0 packets from the interfaces. Before removing the netgraph hooks, the only traffic I seen on any of the three interfaces, was the DHCP request on the ngeth0 virtual interaface, and the bridged ONT_IF interface. So the DHCP requests are still getting to the correct interface.
The fact that tcpdump doesn't see any traffic makes me think its being filtered, like promisc mode isn't allowing EAPOL 802.1X traffic to be capture, and there fore is not bridged. No authentication mean no DHCP response. IMO
I've moved to inline behind the gateway until this can be figured out. I would be willing to test once a day.
Okay, had some success today based on info I gathered from all the various discussions online. I think it is something to do with the em(4) driver. Do all of you having issues have Intel NIC's? I put together a test pfSense server from a bunch of spare parts and it worked right away on the latest release. After digging, I couldn't get any Intel NIC to work. Using what I had around (a few crappy USB dongles worked and old PC's with integrated NICs) I had success with everything not Intel GbE. When I re-upgraded my main pfSense box I was able to move my WAN link to an SFP slot (with RJ45 Module) with some success. I say "some" because all my SFP/RJ45 modules are 10GB and they do not negotiate well with the ONT.
Something interesting for me, if_em.ko is present in /boot/kernel on 2.6.0 but wasn't in my previous version of pfSense. My knowledge is limited but I am not sure where the driver was located in the previous version? Anyone smarter than me know?
Some Useful Links: FreeBSD 12.3 Release Notes (em(4) driver notes) - https://www.freebsd.org/releases/12.3R/relnotes/ Reddit Discussion - https://www.reddit.com/r/PFSENSE/comments/ssgsha/psa_260_breaks_att_bypass/?sort=new Netgate Forum Discussion - https://forum.netgate.com/topic/99190/att-uverse-rg-bypass-0-2-btc/396?_=1644931323812 OPNSense GIT Issue - https://github.com/MonkWho/pfatt/issues/65
I think this is going somewhere because I've tried multiple different boxes but they're all Intel Nics, when I get off work I will try a couple USB dongle's to see if it gets traffic that way.
I think this is going somewhere because I've tried multiple different boxes but they're all Intel Nics, when I get off work I will try a couple USB dongle's to see if it gets traffic that way.
The USBs work for me but are slow. Download is like 100m, upload is better at around 400m. I have a 1G SFP that should get here tomorrow. Really hoping that talks better with the ONT then the 10G did.
For USBs to work at 1 gig speeds you have to have 3.1 USB port or better. For FreeBSD, I am using a box equivalent to the netgate 1541 Same everything but a lot more powerful. Let me know how it goes with the other Nics.
For USBs to work at 1 gig speeds you have to have 3.1 USB port or better. I am using a box equivalent to the netgate 1541 Same everything but a lot more powerful. Let me know how it goes with the other Nics.
Will do! If it helps I'm using the XG-1537 so USB3.0
Is the usb dongles 3.0, when I was using usb in past it worked great I was able to get full 1gb speeds out of my usb ports. If the usb is 3.0 then I don't know why I am getting full 1gb speeds. But I did downgrade back to 2.5.2 now WireGuard don't work on 2.5.2.
Is the usb dongles 3.0
Yep!
Okay, had some success today based on info I gathered from all the various discussions online. I think it is something to do with the em(4) driver.
Nothing useful to add here but I can confirm I'm using an Intel NIC with the em driver. Neither tethered or supplicant working for me on 22.1 but supplicant is working on 21.7.8
em0: <Intel(R) 82583V> port 0xe000-0xe01f mem 0xdf500000-0xdf51ffff,0xdf520000-0xdf523fff irq 16 at device 0.0 on pci1 em1: <Intel(R) 82583V> port 0xd000-0xd01f mem 0xdf400000-0xdf41ffff,0xdf420000-0xdf423fff irq 17 at device 0.0 on pci2 em2: <Intel(R) 82583V> port 0xc000-0xc01f mem 0xdf300000-0xdf31ffff,0xdf320000-0xdf323fff irq 18 at device 0.0 on pci3 em3: <Intel(R) 82583V> port 0xb000-0xb01f mem 0xdf200000-0xdf21ffff,0xdf220000-0xdf223fff irq 19 at device 0.0 on pci4 em4: <Intel(R) 82583V> port 0xa000-0xa01f mem 0xdf100000-0xdf11ffff,0xdf120000-0xdf123fff irq 16 at device 0.0 on pci5 em5: <Intel(R) 82583V> port 0x9000-0x901f mem 0xdf000000-0xdf01ffff,0xdf020000-0xdf023fff irq 17 at device 0.0 on pci6
I'm on a Protectli FW6D
I am using an Intel NIC but with the IGB driver.
I am using an Intel NIC but with the IGB driver.
And is it working or not because my system is using igb drivers too and mine is not working
My knowledge on FreeBSD is limited but I believe igb uses the em(4) driver. All the common Intel cards fall under it (I350, 82575, etc.)
I am using an Intel NIC but with the IGB driver.
And is it working or not because my system is using igb drivers too and mine is not working
Not working
If you look at the if_igb.ko driver in /boot/kernel it just is a shortcut to if_em.ko. I think at one point the two intel drivers merged. https://www.intel.com/content/www/us/en/download/15187/intel-network-adapter-gigabit-base-driver-for-freebsd.html?wapkw=i350%20freebsd
Okay, I got everything up and working on my regular Intel NIC. I’m not the biggest expert here so bear with me.
Through troubleshooting I was able to get every non-Intel NIC to authenticate and pull DHCP. After more testing all igb(4) driver-based cards failed. In the /boot/kernel folder I noticed if_igb.ko is simply a shortcut to the em(4) driver (if_em.ko). I am guessing FreeBSD is using this combined driver from intel? https://www.intel.com/content/www/us/en/download/15187/intel-network-adapter-gigabit-base-driver-for-freebsd.html
Alternatively, I found this driver that appears to be for igb(4) separately, and it seems newer. https://www.intel.com/content/www/us/en/download/14610/intel-network-adapter-driver-for-82575-6-and-82580-based-gigabit-network-connections-under-freebsd.html?wapkw=i350%20freebsd
I downloaded a FreeBSD-12.3 VM, its related source code (amd64), and complied the separate igb(4) driver.
I loaded my newly compiled if_igb.ko into the /boot/modules folder with chmod 555 permissions. Next, I added the following two lines to my /boot/loader.conf file to supersede the included driver.
if_igb_load="YES" if_igb_name="/boot/modules/if_igb.ko"
Rebooted and everything came up just fine!
Feel free to use my compiled if_igb.ko if you don’t want to build your own. https://github.com/neydah700/pfsense_intel/blob/main/if_igb.ko
Also, for reference here is my pfatt script if anyone needs a reference. https://github.com/neydah700/pfsense_intel/blob/main/pfatt_intel.sh
A few notes:
Interesting that the intel igb driver works. I searched for bugs on the FreeBSD buglist and found this...
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260068
Looks like it might be related? Issues with vlan tagging. Was introduced in 13.0 and 12.3... recently fixed in the stable branches, so the timing lines up.
Some comments and feedback in testing so far:
It seems safe to install and test this on 2.5.2. I have downloaded the kernel module and am testing prior to any updates. I haven't managed to break 2.5.2... yet.
It would be better to create /boot/loader.conf.local instead of /boot/loader.conf. Loader.conf may be overwritten by pfsense updates.
What is your output on 2.6.0 with the if_igb.ko module for "kldstat -v"? I can't confirm it is loading and in use on 2.5.2. I am reluctant to upgrade until I can validate it is loading.
Interesting that the intel igb driver works. I searched for bugs on the FreeBSD buglist and found this...
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260068
Looks like it might be related? Issues with vlan tagging. Was introduced in 13.0 and 12.3... recently fixed in the stable branches, so the timing lines up.
Could explain why we are passing 802.1x not pulling DHCP on VLAN 0. I'll add it to my redmine issue on pfSense. If anyone else has success can they go on and comment. Hopefully we get some traction! https://redmine.pfsense.org/issues/12821?next_issue_id=12820
I am testing now reimaging since wiregraud is broke in my install right now.
i am testing now reimaging since wiregraud is broke in my install right now.
Interesting that the intel igb driver works. I searched for bugs on the FreeBSD buglist and found this... https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260068 Looks like it might be related? Issues with vlan tagging. Was introduced in 13.0 and 12.3... recently fixed in the stable branches, so the timing lines up.
Could explain why we are passing 802.1x not pulling DHCP on VLAN 0. I'll add it to my redmine issue on pfSense. If anyone else has success can they go on and comment. Hopefully we get some traction! https://redmine.pfsense.org/issues/12821?next_issue_id=12820
will do internet going out for a bit to update and bring system online.
Some comments and feedback in testing so far:
- It seems safe to install and test this on 2.5.2. I have downloaded the kernel module and am testing prior to any updates. I haven't managed to break 2.5.2... yet.
- It would be better to create /boot/loader.conf.local instead of /boot/loader.conf. Loader.conf may be overwritten by pfsense updates.
- What is your output on 2.6.0 with the if_igb.ko module for "kldstat -v"? I can't confirm it is loading and in use on 2.5.2. I am reluctant to upgrade until I can validate it is loading.
Good point on the .local, will adjust that.
For my kldstat does just this portion work for ya or do you want the whole output?
3 1 0xffffffff83cfb000 35e08 if_igb.ko (/boot/modules/if_igb.ko) Contains modules: Id Name 2 pci/igb
Good point on the .local, will adjust that.
For my kldstat does just this portion work for ya or do you want the whole output?
3 1 0xffffffff83cfb000 35e08 if_igb.ko (/boot/modules/if_igb.ko) Contains modules: Id Name 2 pci/igb
Thank you. That is what I was curious about. It isn't loading on 2.5.2, but it may just be because it was compiled for a different kernel. I also can't get into load with kldload on 2.5.2.
Good point on the .local, will adjust that. For my kldstat does just this portion work for ya or do you want the whole output? 3 1 0xffffffff83cfb000 35e08 if_igb.ko (/boot/modules/if_igb.ko) Contains modules: Id Name 2 pci/igb
Thank you. That is what I was curious about. It isn't loading on 2.5.2, but it may just be because it was compiled for a different kernel. I also can't get into load with kldload on 2.5.2.
I built it using 12.3-RELEASE on amd64 architecture. Hopefully that helps!
Good point on the .local, will adjust that. For my kldstat does just this portion work for ya or do you want the whole output? 3 1 0xffffffff83cfb000 35e08 if_igb.ko (/boot/modules/if_igb.ko) Contains modules: Id Name 2 pci/igb
Thank you. That is what I was curious about. It isn't loading on 2.5.2, but it may just be because it was compiled for a different kernel. I also can't get into load with kldload on 2.5.2.
I built it using 12.3-RELEASE on amd64 architecture.
Time to cross fingers and see how this goes....
Ok, some lessons learned.
pfSense doesn't have wget or curl packages installed by default I didn't want to start mucking up the default packages. It does have fetch. However for whatever reason, when I used fetch to grab the module it was corrupted. After downloading the module on a desktop and uploading it to the firewall with Cyberduck, this worked perfectly. It might work in 2.5.2, but now I have no way of testing. I am now going to try to upgrade to pfSense Plus. I will report. Thank you!
Upgrade to 22.01 from 2.6.0 was perfectly smooth with this workaround in place.
My recommendation to anyone on CE 2.5.2 is to download the module on a desktop and then upload it to your firewall since I encountered issues fetching directly to pfSense. Set up your loader.conf.local as described above. The upgrade to 2.6.0 should then proceed without issue and eventually come back up with a successfully bypassed AT&T gateway. Based on the fix being related to the Intel kernel driver not properly tagging VLAN through netmap, I think this will resolve issues for tethered or supplicant methods. If you wish to register and upgrade to Plus, you can also now do that.
Upgrade to 22.01 from 2.6.0 was perfectly smooth with this workaround in place.
My recommendation to anyone on CE 2.5.2 is to download the module on a desktop and then upload it to your firewall since I encountered issues fetching directly to pfSense. Set up your loader.conf.local as described above. The upgrade to 2.6.0 should then proceed without issue and eventually come back up with a successfully bypassed AT&T gateway. Based on the fix being related to the Intel kernel driver not properly tagging VLAN through netmap, I think this will resolve issues for tethered or supplicant methods. If you wish to register and upgrade to Plus, you can also now do that.
@jasonsansone Glad you are back up! If you get a chance do you mind adding a comment over on the Redmine issue? Hopefully we can get this fixed natively! https://redmine.pfsense.org/issues/12821?next_issue_id=12820
Does everyone who this has worked for extract their own certs?
I noticed you are using the EAP_Identity to apply to the MAC address on ngeth0.
Before when this was working for me I would have to apply the RG MAC to that but then use the MAC that came with the certs in the EAP_Identity part.
Does everyone who this has worked for extract their own certs?
I noticed you are using the EAP_Identity to apply to the MAC address on ngeth0.
Before when this was working for me I would have to apply the RG MAC to that but then use the MAC that came with the certs in the EAP_Identity part.
Ah good point. My EAP and RG MAC are the same so I simplified my script. If someone uses different ones they may need to tailor my script a bit.
I never even looked at the new script. I’m using the same I always have. The newly compiled module was all I needed.
@jasonsansone would you mind posting your script, I cant seem to get mine working with my probably incorrect additions
I’m using the tether method. I never bothered to mess with extracting certificates when I set this up years ago and having the gateway sit there didn’t bother me.
I’m using the tether method. I never bothered to mess with extracting certificates when I set this up years ago and having the gateway sit there didn’t bother me.
The last update broke wpa_supplicant (luckily fixed) and this release broke the VLAN 0 piece. I am strongly considering going back to the tether method to reduce customizations.
Interesting, the tether worked but only after i did the interface assignment from the console, using the assignments within the UI didn't help and then I had to reboot.
Now that ngeth0 is assigned I am going to try and flip the script back to the intel script and see if it works.
@bigjohns97 if you want, I just made a commit to my script to break out the RG and EAP MAC addresses. I haven't tested it but it should work.
https://github.com/neydah700/pfsense_intel/blob/main/pfatt_intel.sh
@neydah700 Thanks for figuring out the fix. I posted to my website to hopefully help people find the fix faster. Please let me know if you think anything should be added or removed to it. https://angrysysadmins.tech/index.php/2022/02/grassyloki/pfsense-2-6-0-fix-pf-att-bypass-mode/
@neydah700 Thanks for figuring out the fix. I posted to my website to hopefully help people find the fix faster. Please let me know if you think anything should be added or removed to it. https://angrysysadmins.tech/index.php/2022/02/grassyloki/pfsense-2-6-0-fix-pf-att-bypass-mode/
Thanks for sharing! One note, while this seems to be fixing the ibg(4) people I am guessing it is not fixing the em(4) people. There is more discussion going on over at the Netgate Forums. I am going to compile the combined driver and see if that still works for me. So maybe will need an update if this works!
@bigjohns97 if you want, I just made a commit to my script to break out the RG and EAP MAC addresses. I haven't tested it but it should work.
https://github.com/neydah700/pfsense_intel/blob/main/pfatt_intel.sh
Thanks for everything, unfortunately this didn't work for me but it did tell me my script was correct, the only difference between yours and mine was mine had an extra space on the logger line.
I guess I am stuck running tether mode from now on until someone maybe can solve this issue I had.
Maybe I go and try to extract the actual certs from my device at a later date and see if it's the certs that are bad or if they aren't allowing the interface MAC to not match the EAP_Identity MAC now or something.
@bigjohns97
Sorry about that! for what it's worth I have never used anything but the same MAC's, so my script might be wrong. I guess only other suggestion is make sure pfSense is not managing that interface and that you set the ngeth0 MAC address to match your RG MAC in the web interface?
@bigjohns97
Sorry about that! for what it's worth I have never used anything but the same MAC's, so my script might be wrong. I guess only other suggestion is make sure pfSense is not managing that interface and that you set the ngeth0 MAC address to match your RG MAC in the web interface?
I tried adding this but it didn't make a difference, tether was working without it as well so I am hanging it up for the day until I get get some time to get those certs out of the RG.
Thanks again @neydah700
i got the certs out again and now it don't work I cant get pass the
WPA_STATUS_CMD="wpa_cli status | grep 'suppPortStatus' | cut -d= -f2" IP_STATUS_CMD="ifconfig ngeth0 | grep 'inet\ ' | cut -d' ' -f2" /usr/bin/logger -st "pfatt" "waiting for EAP authorization..."
During all this messing. I deleted my cert, so I had to pull again.
=Here is my full script
EAP_SUPPLICANT_IDENTITY="" RG_ETHER_ADDR="" LOG=/var/log/pfatt.log ONT_IF="igb0"
getTimestamp(){
echo date "+%Y-%m-%d %H:%M:%S :: [pfatt.sh] ::"
}
/usr/bin/logger -st "pfatt" "starting pfatt..." /usr/bin/logger -st "pfatt" "configuration:" /usr/bin/logger -st "pfatt" " ONT_IF = $ONT_IF" /usr/bin/logger -st "pfatt" " EAP_SUPPLICANT_IDENTITY = $EAP_SUPPLICANT_IDENTITY" /usr/bin/logger -st "pfatt" " RG_ETHER_ADDR = $RG_ETHER_ADDR"
/usr/bin/logger -st "pfatt" "resetting netgraph..." /usr/sbin/ngctl shutdown $ONT_IF: >/dev/null 2>&1 /usr/sbin/ngctl shutdown vlan0: >/dev/null 2>&1 /usr/sbin/ngctl shutdown ngeth0: >/dev/null 2>&1
/usr/bin/logger -st "pfatt" "your ONT should be connected to pyshical interface $ONT_IF" /usr/bin/logger -st "pfatt" "creating vlan node and ngeth0 interface..." /usr/sbin/ngctl mkpeer $ONT_IF: vlan lower downstream /usr/sbin/ngctl name $ONT_IF:lower vlan0 /usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether /usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }' /usr/sbin/ngctl msg ngeth0: set $RG_ETHER_ADDR
/usr/bin/logger -st "pfatt" "enabling promisc for $ONT_IF..." /sbin/ifconfig $ONT_IF ether $RG_ETHER_ADDR /sbin/ifconfig $ONT_IF up /sbin/ifconfig $ONT_IF promisc
/usr/bin/logger -st "pfatt" "starting wpa_supplicant..."
WPA_PARAMS="\ set eapol_version 2,\ set fast_reauth 1,\ ap_scan 0,\ add_network,\ set_network 0 ca_cert \\"/root/pfatt/wpa/ca.pem\\",\ set_network 0 client_cert \\"/root/pfatt/wpa/client.pem\\",\ set_network 0 eap TLS,\ set_network 0 eapol_flags 0,\ set_network 0 identity \\"$EAP_SUPPLICANT_IDENTITY\\",\ set_network 0 key_mgmt IEEE8021X,\ set_network 0 phase1 \\"allow_canned_success=1\\",\ set_network 0 private_key \\"/root/pfatt/wpa/private.pem\\",\ enable_network 0\ " WPA_DAEMON_CMD="/usr/sbin/wpa_supplicant -Dwired -i$ONT_IF -B -C /var/run/wpa_supplicant"
PID=$(pgrep -f "wpa_supplicant") if [ ${PID} > 0 ]; then /usr/bin/logger -st "pfatt" "terminating existing wpa_supplicant on PID ${PID}..." RES=$(kill ${PID}) fi
RES=$(${WPA_DAEMON_CMD}) PID=$(pgrep -f "wpa_supplicant") /usr/bin/logger -st "pfatt" "wpa_supplicant running on PID ${PID}..."
/usr/bin/logger -st "pfatt" "setting wpa_supplicant network configuration..." IFS="," for STR in ${WPA_PARAMS}; do STR="$(echo -e "${STR}" | sed -e 's/^[[:space:]]*//')" RES=$(eval wpa_cli ${STR}) done
WPA_STATUS_CMD="wpa_cli status | grep 'suppPortStatus' | cut -d= -f2" IP_STATUS_CMD="ifconfig ngeth0 | grep 'inet\ ' | cut -d' ' -f2" /usr/bin/logger -st "pfatt" "waiting for EAP authorization..."
i=1 until [ "$i" -eq "5" ] do sleep 5 WPA_STATUS=$(eval ${WPA_STATUS_CMD}) if [ X${WPA_STATUS} = X"Authorized" ]; then /usr/bin/logger -st "pfatt" "EAP authorization completed..."
IP_STATUS=$(eval ${IP_STATUS_CMD})
if [ -z ${IP_STATUS} ] || [ ${IP_STATUS} = "0.0.0.0" ];
then
/usr/bin/logger -st "pfatt" "no IP address assigned, force restarting DHCP..."
RES=$(eval /etc/rc.d/dhclient forcerestart ngeth0)
IP_STATUS=$(eval ${IP_STATUS_CMD})
fi
/usr/bin/logger -st "pfatt" "IP address is ${IP_STATUS}..."
/usr/bin/logger -st "pfatt" "ngeth0 should now be available to configure as your WAN..."
sleep 5
/usr/bin/logger -st "pfatt" "set mac address on ngeth0..."
/sbin/ifconfig ngeth0 ether $RG_ETHER_ADDR
break
else
/usr/bin/logger -st "pfatt" "no authentication, retrying ${i}/5..."
i=$((i+1))
fi
done
@SGC1990 what folder are your certs in and do they have the appropriate permissions? That script assumes they are in the /root/pfatt/wpa/ directory.
also, are you using igb0?
The dhcp lease for connections is not handed through to the ngeth0 interface properly. There isn't any real "errors" in the logs.
If you try to run the script manually after boot you get "ngctl: send msg: File exists"
Logs from pfatt.log:
2022-02-14 14:36:56 :: [pfatt.sh] :: pfSense + AT&T U-verse Residential Gateway for true bridge mode 2022-02-14 14:36:56 :: [pfatt.sh] :: Configuration: 2022-02-14 14:36:56 :: [pfatt.sh] :: ONT_IF: igb0 2022-02-14 14:36:56 :: [pfatt.sh] :: RG_IF: igb1 2022-02-14 14:36:56 :: [pfatt.sh] :: RG_ETHER_ADDR: [MY MAC HERE] 2022-02-14 14:36:56 :: [pfatt.sh] :: attaching interfaces to ng_ether... OK! 2022-02-14 14:36:56 :: [pfatt.sh] :: building netgraph nodes... 2022-02-14 14:36:56 :: [pfatt.sh] :: creating ng_one2many... 2022-02-14 14:37:00 :: [pfatt.sh] :: pfSense + AT&T U-verse Residential Gateway for true bridge mode
I am not running wpa_supplicant mode.