opnsense on FW4B Supplicant

[wsciaroni]

Hello,

I've been troubleshooting this for several days now and am finally coming with an issue. I'd be happy to provide more info.

I have previously used bridge mode with pfsense at two other installs without any issues! Thank you for all the work you put into this project.

What I'm working with

However, I have migrated to opnsense when I moved and I'm having trouble. Here's what I'm using:

Software

OPNsense 21.1.4-amd64
FreeBSD 12.1-RELEASE-p15-HBSD
OpenSSL 1.1.1k 25 Mar 2021

Hardware

Protectli FW4B (With 4g failover Nic shows as a USB device ue0)

BGW210-700 XSGPON ONT (one of the two sites I've done this previously had the same ONT)

What I'm seeing

• I've got the certs and I'm using the mac from the certs.
• I've done packet captures and made sure that I've entered the correct EAP identity in the script
• I've tried matching permissions of the certs with no luck there either. I've tried 600 666 and 700 as recommended.

When I take a capture of the [RG] -- [ONT] traffic I am able to capture the entire EAP handshake. I noticed that none of the EAP traffic are tagged on vlan0.

However, when I attempt [opnsense] -- [ont] and capture that traffic, my EAP outbound start packet is being tagged on vlan0. This is causing a loop in which both devices are ready to start authentication, but I'm on vlan0 and they're waiting around looking for untagged traffic.

At least that's what I suspect is happening as it's the only difference I can really tell. So, essentially I'm never seeing the EAP traffic progress past a EAP Request Identity and EAP Start.

Additional remarks:

Packet Capture Method

I wanted to add that I'm capturing the packets using a switch set up to mirror 3 ports. In this way I have the RG, ONT, and capture interface on laptop tied to the switch when capturing [RG] -- [ONT]. I have ONT, Opnsense, and capture interface on laptop tied to the switch when capturing [Opnsense]--[ONT].

What I'm capturing

The actual EAPOL start packet that I captured can be seen below. Clearly somehow it is getting tagged...

When it's [RG]--[ONT] I see the following in the start frame (Notice no tagging)

I am able to observe the entire EAP handshake for [RG]--[ONT]

Absolutely forgot to mention that I'm using the opnatt.sh from the supplicant_OPNsense_testing branch.Absolutely forgot to mention that I'm using the opnatt.sh from the supplicant_OPNsense_testing branch.

Conclusion

Thoughts?

[cspencer49519]

Have you tried manually executing opnatt.sh after the device has fully booted? I haven't logged it, but I'm on one version previous to you:

OPNsense 21.1.3_3-amd64FreeBSD 12.1-RELEASE-p14-HBSDOpenSSL 1.1.1j 16 Feb 2021

Since I updated to this version, anytime I reboot my router I need to execute the script manually after reboot to get everything up and running.

[wsciaroni]

I have manually run the script as well as run each line of the script manually. Once I get the script running I see that the handshake is being vlan tagged where it should not be. I'm not sure why this is happening. It is breaking the negotiation.

[zombielinux]

Where exactly in the script is this happening? I'm having a similar issue thats been occuring for the same time period.

[wsciaroni]

@zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be.

[zombielinux]

@zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be.

Sorry this took a minute to get back to. Take a look at my modified opnsense.sh script in my pull request. See if that works for you.

Also, if you DID get it working, did IPV6 work for you out of the box? because it isn't for me. See here: https://github.com/opnsense/core/issues/5474

[wsciaroni]

@zombielinux The script appears to complete execution successfully. In fact, I ran the commands individually and saw the same behavior. I don't see any error codes during the script. I simply see that the authentication is tagged when it shouldn't be.

Sorry this took a minute to get back to. Take a look at my modified opnsense.sh script in my pull request. See if that works for you.

Also, if you DID get it working, did IPV6 work for you out of the box? because it isn't for me. See here: https://github.com/opnsense/core/issues/5474

Unfortunately, this still does not work for me. I'll have to look at a packet capture later... But, I suspect it's the same root cause.

I have a question. Is your authentication traffic vlan tagged? I'm curious about the timing of the enabling of the vlan0.

Thanks!

[zombielinux]

It should be. Are you pointing it at the right interface? You can try running it line by line and see what errors it spits out.

What ONT do you have?

[wsciaroni]

I tried running it line by line with no success. I triple checked that I'm using the right interface id.

I noticed that the method used to determine the PID is not finding anything. When I run it without the .ngeth part it returns a pid.

I don't have the model of the ONT on hand, but it's the flat black one that is not wall mounted (don't know if that means anything)...

I can say more later when I'm with the box. Thanks for the help. Feel free to provide any suggestions that I can try out later. Thank you

[zombielinux]

That sounds like a nokia 020.

I will say that my WAN interface after the fact isn't an ngeth0 for whatever reason, but the raw interface (em1 in my case).

[wsciaroni]

It is a Nokia 020.

So, you ended up setting your wan to the same interface ONT_IF rather than ngeth0?

[zombielinux]

Yep. Its been working fine for a while now (with the exception of ipv6)

[wsciaroni]

Hmm... I re-checked. I had mis-typed my mac address......

However, once corrected, I see the same behavior (Still haven't done a packet capture though).

I see the following output running opnatt-supplicant.sh as root

pfatt 57748 - - starting pfatt...
pfatt 82551 - - configuration:
pfatt 97447 - -   ONT_IF = igb0
pfatt 31176 - -   RG_ETHER_ADDR = XX:XX:XX:XX:XX:XX
pfatt 50916 - -   EAP_MODE = supplicant
pfatt 78201 - -   EAP_SUPPLICANT_IDENTITY = XX:XX:XX:XX:XX:XX
pfatt 11298 - - resetting netgraph...
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
ngctl: shutdown: No such file or directory
pfatt 10769 - - configuring EAP environment for supplicant mode...
pfatt 24064 - - cabling should look like this:
pfatt 66080 - -   ONT---[] [igb0]HOSTNAME...
pfatt 98464 - - creating vlan node and ngeth0 interface...
pfatt 24250 - - enabling promisc for igb0...
pfatt 43291 - - starting wpa_supplicant...
pfatt 73055 - - wpa_supplicant running on PID 33428...
pfatt 81811 - - setting wpa_supplicant network configuration...
pfatt 1633 - - waiting EAP for authorization...

With the wpa_supplicant config:

eapol_version=1
ap_scan=0
fast_reauth=1
network={
    ca_cert="/conf/pfatt/wpa/ca.pem"
    client_cert="/conf/pfatt/wpa/client.pem"
    eap=TLS
    eapol_flags=0
    identity="XX:XX:XX:XX:XX:XX"
    key_mgmt=IEEE8021X
    phase1="allow_canned_success=1"
    private_key="/conf/pfatt/wpa/private.pem"
}

This is modifying the PID command to PID=$(pgrep -f "wpa_supplicant")

[zombielinux]

Is your ONT_IF really igb0?

And is XX:XX:XX:XX:XX:XX the mac address that you're spoofing?

Have you also set you wan interface to spoof that address via the GUI?

[wsciaroni]

Yes, the port labeled WAN on my Protectli FW4B is igb0

I didn't want to include the MAC from my RG, but yes. I replaced it in the logs for privacy. That is the mac of my RG.

Yes, I have that interface set to spoof that address via the GUI as well.

[zombielinux]

OH! I remember whats happening. The ngctl commands that output to >/dev/null 2>&1do NOT like the bash implementation on opnsense. I'm not sure why. But those "not founds" are correct.

What you CAN do is try power cycling the whole thing, including the ONT. It took some fiddling for me to get it right, but its been rock solid ever since.

[wsciaroni]

@zombielinux Could you help me modify the script to not start tagging the vlan until after EAP authentication has succeeded? I think that may be what's causing my issue.

[zombielinux]

@zombielinux Could you help me modify the script to not start tagging the vlan until after EAP authentication has succeeded? I think that may be what's causing my issue.

As far as I know, the EAP authentication has to be done over the tagged interface, but after that, nothing has to be tagged to communicate.

VLAN0 is actually more of a QoS tagging, rather than a traditional VLAN

[wsciaroni]

After all I've read I came to the same understanding. But based on my packet capture of my particular RG/ONT authentication, the EAP traffic is not tagged (see above). Maybe I need to modify the script to not tag anything with vlan0.

[zombielinux]

Give that a try. I vaguely remember that working on pfsense a few years ago.

On Sun, Jan 30, 2022 at 12:08 Will Sciaroni @.***> wrote:

After all I've read I came to the same understanding. But based on my packet capture of my particular RG/ONT authentication, the EAP traffic is not tagged (see above). Maybe I need to modify the script to not tag anything.

— Reply to this email directly, view it on GitHub https://github.com/MonkWho/pfatt/issues/53#issuecomment-1025186776, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABN74JXDVCGOIVQCF32UELUYVWATANCNFSM42NSFINA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- No trees were harmed in the sending of this message, but a rather large number of electrons were terribly inconvenienced.