search

MonkWho / pfatt

Enable true bridge mode for AT&T U-Verse and pfSense (this is a fork of an original repository https://github.com/aus/pfatt. Since it is not available anymore, I'll do my best to maintain a copy for people that still need a bypass)
440 stars 171 forks source link

OPNSense 20.7 and OPNatt.sh: No DHCP from ATT #17

Closed lrosenman closed 3 years ago

lrosenman commented 3 years ago

I have this working perfectly on 20.1, but if I upgrade to 20.7, I don't get addresses from ATT.

Anyone try it on 20.7 yet?

MonkWho commented 3 years ago

I personally don't use OPNSense so I can't comment. I can only recommend you take your script and add a bunch of logging and pauses to it so when your firewall is booting and script is running for the first time you can see what happens at each step and try to figure out where it's breaking.

MonkWho commented 3 years ago

Can you please try running "kldstat -v" and see what comes up. Make sure that these are loaded: netgraph ng_ether ng_eiface ng_one2many ng_vlan ng_etf

When OPNSense 20.1 came out we had to add a section to the script that made sure those modules were loaded. And I have no idea what they changed in 20.7 that could be causing this.

maxfield-allison commented 3 years ago

Looks like everything is following the path you laid out in troubleshooting steps in the readme. I'm seeing eapol start on my ont interface followed by DHCP requests but when I run tcpdump for port 67/68 on ONT I'm not seeing any traffic. Opnsense general logs state that no DHCP offer is received despite many DHCP discover packets being transmitted on ngeth0

maxfield-allison commented 3 years ago

To clarify, all of the needed modules load and traffic is being sent correctly for the most part. I'm just not seeing anything leave the ONT interface on ports 67/68. I'm going to workaround and plug back in the default way to bring my workstation back online so I don't have to keep posting from my phone lol

maxfield-allison commented 3 years ago

Here's the output for tcpdump on each interface: RG_IF

listening on igb1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:57:10.927702 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e),                                                                                              length 60: EAPOL start (1) v2, len 0
18:57:42.156973 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
18:58:13.382699 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
18:58:44.607537 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

ONT_IF

listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:00:18.282420 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
19:00:49.507720 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
19:01:11.920184 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:01:18.027119 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:01:20.733007 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
19:01:30.068489 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:01:51.061137 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:01:51.958344 ec:a9:40:c3:20:d1 (oui Unknown) > 01:80:c2:00:00:03 (oui Unknown), ethertype EAPOL (0x888e), length 60: EAPOL start (1) v2, len 0
19:02:06.182348 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 346: vlan 0, p 0, ethertype IPv4, OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel

ngeth0

listening on ngeth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:06:08.968794 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:08.968876 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:16.093504 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:16.093589 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:26.002201 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:26.002293 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:39.011289 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
19:06:39.011397 ec:a9:40:c3:20:d1 (oui Unknown) > Broadcast, ethertype IPv4 (0x0800), length 342: OPNsense.local.bluewillows.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from ec:a9:40:c3:20:d1 (oui Unknown), length 300
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

And finally, What I'm seeing in OPNsense logs

2020-08-01T19:06:26 | dhclient[75651]: DHCPDISCOVER on ngeth0 to 255.255.255.255 port 67 interval 13
2020-08-01T19:06:16 | dhclient[75651]: DHCPDISCOVER on ngeth0 to 255.255.255.255 port 67 interval 10
2020-08-01T19:06:08 | dhclient[75651]: DHCPDISCOVER on ngeth0 to 255.255.255.255 port 67 interval 8

followed shortly after by a message indicating no offer was received.

maxfield-allison commented 3 years ago

@MonkWho is there any information I can provide or steps I can take so we can narrow down what's going on? I'm going to open an issue with a request for more information on what may have changed in the latest OPNsense release on their github.

maxfield-allison commented 3 years ago

want to keep my finger on the pulse here. I have no problem testing things out and have begun to look into what changed in OPNsense. I'll update with my findings shortly but would love input on anything I may be able to try in the interim.

chenxiaolong commented 3 years ago

Just adding a data point to this thread, the wpa_supplicant method (which still uses netgraph for vlan 0) is working fine in OPNsense 20.7 for me. I assume whatever the problem is, it only affects the way netgraph is set up for the "normal" method.

maxfield-allison commented 3 years ago

Adding this thread from the OPNsense forums for your perusal @MonkWho. I plan on trying again later in the week doing some of the troubleshooting steps included in this thread e.g. disabling IPS.

MonkWho commented 3 years ago

Thanks for adding more info to the issue @maxfield-allison. Good luck with your tests and hope things work in your new install.

MonkWho commented 3 years ago

There hasn't been any activity in this thread for a while so I'm going to go ahead and close the issue.

maxfield-allison commented 3 years ago

I've finally found some time to take a deeper look at this. Between my last update and now I've also installed the netmap kernel that fixes a host of issues with the drivers for my NIC's (em and igb drivers). Even with adding the required modules to bootloader.local I don't see all of the required modules loaded when I do kldstat -v. I'm thinking I may go ahead and roll back to the included kernel to see if I can get this sorted on that first but alternatively, I think I'm going to contact the developer of the new netmap kernel and see if I can get them to provide these modules in the repo even if they don't install by default. Once I can get a better handle on this, I may try to put together this script in a package that can be hosted on the OPNsense repo and installed via the GUI and a simple wizard.

maxfield-allison commented 3 years ago

Interestingly enough, the boot/kernel folder still contains the modules in question. So it's not that they don't exist but I can't figure out how to get them loaded.

maxfield-allison commented 3 years ago

manual kldload works to load almost all of the required modules. they just wont load from loader.conf.local

dkowis commented 3 years ago

I just upgraded to 20.7, and I have all the modules loaded, so far, everything is working except ipv6.

maxfield-allison commented 3 years ago

@MonkWho So how about this shit. I removed the set -e option in the opnatt.sh script and boom, everything works. I went as far as to try to get the supplicant method to work (I'm still aiming for it, already pulled my RG certs) and decided to try the bridge mode again on a whim. ran it manually from the CLI and got invalid argument set-. Pulled that out of the script and viola. I'll submit a pull request with changes once I confirm nothing else is screwy. Also intend to test and confirm supplicant methods and scripts and pull request all of that along with documentation as soon as I can. The wife is going to put me down if I work on this any longer tonight ;]

maxfield-allison commented 3 years ago

I have this working perfectly on 20.1, but if I upgrade to 20.7, I don't get addresses from ATT.

Anyone try it on 20.7 yet?

try removing set -e from the start of the script and rebooting.

dkowis commented 3 years ago

24 hours after upgrading, with no changes to the pfatt script, and all my internets are working. IPv6 isn't, but everything else is working fine. I think it's probably a firewall settings, since I get an address, but for some reason my computer claims "Network is unreachable" when using ping6.

Yeah, on another host, ping6 works fine, so it's definitely this laptop. So yeah, no changes from the github repo, no set -e removal. Everything just works.

Here's the entire contents, as I've got it on mine. It wouldn't let me attach it :(

#!/bin/sh
set -e

ONT_IF='igb3'
RG_IF='re0'
RG_ETHER_ADDR='MAC_ADDRESS'
OPNSENSE='yes'
LOG=/var/log/pfatt.log

getTimestamp(){
    echo `date "+%Y-%m-%d %H:%M:%S :: [pfatt.sh] ::"`
}

{
    echo "$(getTimestamp) pfSense + AT&T U-verse Residential Gateway for true bridge mode"
    echo "$(getTimestamp) Configuration: "
    echo "$(getTimestamp)        ONT_IF: $ONT_IF"
    echo "$(getTimestamp)         RG_IF: $RG_IF"
    echo "$(getTimestamp) RG_ETHER_ADDR: $RG_ETHER_ADDR"
    echo "$(getTimestamp)      OPNSENSE: $OPNSENSE"

    echo -n "$(getTimestamp) loading netgraph kernel modules... "
    /sbin/kldload -nq ng_etf
    echo "OK!"

    if [ ${OPNSENSE} != 'yes' ]; then
        echo -n "$(getTimestamp) attaching interfaces to ng_ether... "
        /usr/local/bin/php -r "pfSense_ngctl_attach('.', '$ONT_IF');" 
        /usr/local/bin/php -r "pfSense_ngctl_attach('.', '$RG_IF');"
        echo "OK!"
    fi 

    echo "$(getTimestamp) building netgraph nodes..."

    echo -n "$(getTimestamp)   creating ng_one2many... "
    /usr/sbin/ngctl mkpeer $ONT_IF: one2many lower one
    /usr/sbin/ngctl name $ONT_IF:lower o2m
    echo "OK!"

    echo -n "$(getTimestamp)   creating vlan node and interface... "
    /usr/sbin/ngctl mkpeer o2m: vlan many0 downstream
    /usr/sbin/ngctl name o2m:many0 vlan0
    /usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether

    /usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'
    /usr/sbin/ngctl msg ngeth0: set $RG_ETHER_ADDR
    echo "OK!"

    echo -n "$(getTimestamp)   defining etf for $ONT_IF (ONT)... "
    /usr/sbin/ngctl mkpeer o2m: etf many1 downstream
    /usr/sbin/ngctl name o2m:many1 waneapfilter
    /usr/sbin/ngctl connect waneapfilter: $ONT_IF: nomatch upper
    echo "OK!"

    echo -n "$(getTimestamp)   defining etf for $RG_IF (RG)... "
    /usr/sbin/ngctl mkpeer $RG_IF: etf lower downstream
    /usr/sbin/ngctl name $RG_IF:lower laneapfilter
    /usr/sbin/ngctl connect laneapfilter: $RG_IF: nomatch upper
    echo "OK!"

    echo -n "$(getTimestamp)   bridging etf for $ONT_IF <-> $RG_IF... "
    /usr/sbin/ngctl connect waneapfilter: laneapfilter: eapout eapout
    echo "OK!"

    echo -n "$(getTimestamp)   defining filters for EAP traffic... "
    /usr/sbin/ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
    /usr/sbin/ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
    echo "OK!"

    echo -n "$(getTimestamp)   enabling one2many links... "
    /usr/sbin/ngctl msg o2m: setconfig "{ xmitAlg=2 failAlg=1 enabledLinks=[ 1 1 ] }"
    echo "OK!"

    echo -n "$(getTimestamp)   removing waneapfilter:nomatch hook... "
    /usr/sbin/ngctl rmhook waneapfilter: nomatch
    echo "OK!"

    echo -n "$(getTimestamp) enabling $RG_IF interface... "
    /sbin/ifconfig $RG_IF up
    echo "OK!"

    echo -n "$(getTimestamp) enabling $ONT_IF interface... "
    /sbin/ifconfig $ONT_IF up
    echo "OK!"

    echo -n "$(getTimestamp) enabling promiscuous mode on $RG_IF... "
    /sbin/ifconfig $RG_IF promisc
    echo "OK!"

    echo -n "$(getTimestamp) enabling promiscuous mode on $ONT_IF... "
    /sbin/ifconfig $ONT_IF promisc
    echo "OK!"

    echo "$(getTimestamp) ngeth0 should now be available to configure as your pfSense WAN"
    echo "$(getTimestamp) done!"
} >> $LOG
MonkWho commented 3 years ago

If script works when you remove "set -e" then I have no answer. All "-e" does is "Exit immediately if a command exits with a non-zero exit status". And I don't see how that could be causing it not work for you @maxfield-allison.

maxfield-allison commented 3 years ago

I understand. It doesn't make sense at all. I ended up upgrading to 20.7.3 and installing the new netmap kernel but I can't figure out why it stopped working in the first place or why removing that option did anything differently. I did manually create ngeth0 at one point and after that it survived reboots and pulled dhcp. lot's of weird happenings.