Running phantap on other router models

[s0meguy1]

Just a question, could this be run on the GL-AR750S, which is gigabit? Or something a littler faster, like the MT300N?

[champtar]

Hi @s0meguy1

The idea is that you need all the data to go through the linux bridge (br-phantap) for PhanTap to work. Many small devices have an integrated switch with 1 port connected to the CPU. So to make it work you need to configure 2 ports as 2 different VLANs, and put them in the br-phantap bridge (you would have for example eth0.1 and eth0.2).

The limitation is that your device might block tagged traffic (802.1Q), which in most case will be fine (I think most devices don't support double tagging). Also OpenWrt currently use swconfig for switches, but I think they are slowly switching to DSA, which might help. (good presentation about DSA https://netdevconf.org/2.1/papers/distributed-switch-architecture.pdf)

I have a GL-AR750S, I'll try to have a look today.

[s0meguy1]

Thanks! Let me know, I ordered one on Amazon but if it doesn't work, no worries, I can easily return it, after I do some testing trying to get it to work that is

[champtar]

On a GL-AR750S-EXT

root@OpenWrt:~# swconfig dev switch0 help
switch0: mdio-bus.0(Atheros AR8337), ports: 7 (cpu @ 0), vlans: 128
     --switch
    Attribute 1 (int): enable_vlan (Enable VLAN mode)
    Attribute 2 (none): reset_mibs (Reset all MIB counters)
    Attribute 3 (int): enable_mirror_rx (Enable mirroring of RX packets)
    Attribute 4 (int): enable_mirror_tx (Enable mirroring of TX packets)
    Attribute 5 (int): mirror_monitor_port (Mirror monitor port)
    Attribute 6 (int): mirror_source_port (Mirror source port)
    Attribute 7 (int): arl_age_time (ARL age time (secs))
    Attribute 8 (string): arl_table (Get ARL table)
    Attribute 9 (none): flush_arl_table (Flush ARL table)
    Attribute 10 (int): igmp_snooping (Enable IGMP Snooping)
    Attribute 11 (int): igmp_v3 (Enable IGMPv3 support)
    Attribute 12 (none): apply (Activate changes in the hardware)
    Attribute 13 (none): reset (Reset the switch)
     --vlan
    Attribute 1 (int): vid (VLAN ID (0-4094))
    Attribute 2 (ports): ports (VLAN port mapping)
     --port
    Attribute 1 (none): reset_mib (Reset single port MIB counters)
    Attribute 2 (string): mib (Get port's MIB counters)
    Attribute 3 (int): enable_eee (Enable EEE PHY sleep mode)
    Attribute 4 (none): flush_arl_table (Flush port's ARL table entries)
    Attribute 5 (int): igmp_snooping (Enable port's IGMP Snooping)
    Attribute 6 (int): vlan_prio (Port VLAN default priority (VLAN PCP) (0-7))
    Attribute 7 (int): pvid (Primary VLAN ID)
    Attribute 8 (unknown): link (Get port link information)

So you don't have double tagging, but if you just want to use it on a classic ethernet link without VLAN it works fine

It seems there is an upstream DSA driver for this switch, https://lwn.net/Articles/646595/, so someone with more knowledge than me just need to play with the dts and that should work, and then we would not need double tagging to be able to let through tagged packets

[s0meguy1]

Thats pretty good news. Personally I do not know how big my need is for seeing VLAN traffic is, but I'll play around with it, even though attempting to do this may also be above my pay grade. It comes this weekend but I may not be able to play with it until Monday. Thanks for looking into it!

[s0meguy1]

Ok,

I am not as savvy as I thought when it comes to this. How were you able to install it on the 750s? opkg didn't have it in its repos, so I tried to install generic openwrt on it (here) and it bricked it (I had to use sysupgrade -F -n ./file). I had to reflash the stock back. How did you get it working?

[champtar]

Hi @s0meguy1

Sorry for the delay. GL.iNET include an easy to use recovery mode to flash again the router, maybe I had to use that, I don't remember. See the instructions here https://docs.gl-inet.com/en/2/troubleshooting/debrick/, the only difference is that the default IP of OpenWrt is 192.168.1.1

[s0meguy1]

Yeah was able to unbrick it, I just can't get phantap installed on it. What steps did you take to get phantap running?

[champtar]

What error do you have ? What image did you flash ?

[s0meguy1]

So with a fresh boot onto the AR750S, I do an opkg install phantap phantap-learn, and it cannot find the package. I tried to compile from source, however g++/gcc is not supported by openwrt. So I wasn't able to even follow the basic instructions.

[champtar]

Hi,

Did you do opkg update first ? Are you using the master/snapshot images ? (PhanTap is not in the stable branches for now)

[s0meguy1]

So initially I used the default snapshot, that failed, I did some googling and realized I needed the master, as you said. So I went here (https://downloads.openwrt.org/snapshots/targets/ath79/generic/) and downloaded the master, attempted to flash the device via the web GUI and it gave me an error. So then I ssh'ed into the device and flashed it using the CLI, with the commands: sysupgrade -F -n ./file.bin After I ran that, it bricked the device, and I gave up and posted on here

[champtar]

Now that you have the snapshot working (if I understand correctly), have you run opkg update before opkg install ... ?

[s0meguy1]

Yes, I did, however I am unsure if the link I provided was the correct master. Can you confirm that is the master branch (https://downloads.openwrt.org/snapshots/targets/ath79/generic/)?

[champtar]

Looks good to me yes

[s0meguy1]

Here's what I get when I try to upgrade from the web application:

Here's what I get when I try and upgrade from the CLI:

[champtar]

You cannot use the initramfs, you need the sysupgrade file https://downloads.openwrt.org/snapshots/targets/ath79/generic/openwrt-ath79-generic-glinet_gl-ar750s-squashfs-sysupgrade.bin you can also flash this file via the router recovery

[s0meguy1]

I think it worked! I need to confirm when I get home from work because I am not able to connect this device to my work network, but I was successfully able to flash sysupgrade! I'll let you know the results later...

[champtar]

@s0meguy1 everything working now ?

[s0meguy1]

Yes! Thanks for your help - I was just trying to flash the wrong version of openWRT. I am having issues, however it has to do with the fact I have two bridges from setup and not phantap. I haven't played with it since Tues, but I tired to setup a local bride for configurations and I had install phatap, so I also had the phantap bridge. I wanted to access it via wifi while it was MiTM'ing traffic, but I more than likely misconfigured it. I believe the fact there were two bridges messed up the networking. I had planned to re-flash then try again, but work/life got in the way. Monday the 19th I plan on re-flashing, installing everything I need, then installing phantap. I report back after that

[1989gironimo]

@s0meguy1 Do you have an update for us? Interested in GBit eth as well.

[RoganDawes]

In case anyone is interested, I am planning to get phantap working on the NanoPi R1S, which has 1 true GbE interface, and one USB2->GbE interface (i.e. limited to ~300Mbps, but will still negotiate GbE with the switch). So far I have been fighting with FriendlyWrt (FriendlyArm's version of OpenWrt) to build a minimal image, but I expect it to submit soon! :-D The nice thing about the R1S is that it uses entirely separate interfaces, rather than switch VLAN's, as well as being only $20 each. One other thing to consider is that data is stored on uSD, meaning that you can replace uBoot trivially without having to worry about bricking your device, addressing the early boot notification packets. But also that anyone poking around at it can easily examine the filesystem, and see what you are up to.

[champtar]

Hi @RoganDawes, if you write some instructions/blog we can add a link to the README

[RoganDawes]

Will do!

[H8to]

Nice idea with the NanoPi R1S! Eager to see the writeup, too :)

[RoganDawes]

I haven't really made an effort to get phantap working on it so far, I have a shell script that does the necessary quite effectively. My build is based on Armbian, though, rather than OpenWrt, purely because all the devices worked from the outset with Armbian, and I struggled for a long time to get OpenWrt working (and failed).

One interesting thing, though, is that it doesn't seem necessary to patch the kernel to pass 802.1x traffic across the bridge. It just works, when I set bridge/group_forward_mask to 65528. I'm curious to know if I am missing a test case here, though?

A cute trick that I use to prevent leaking of traffic onto the target network is to put the bridge and interfaces into a network namespace. Then only processes deliberately placed in the namespace are able to access the interface.

I also have a python program watching tcpdump output for ARP, DNS, and UDP/TCP packets to update the arp tables, hosts files, and routing tables. It is not currently adding a default gateway if it identifies any non-RFC1918 traffic, but that would be a very simple thing to add. It also runs dnsmasq watching the directory that I write the DNS answers to, so that the user can refer to previously observed hosts by name, without ever putting their own DNS queries onto the wire.

[champtar]

I haven't really made an effort to get phantap working on it so far, I have a shell script that does the necessary quite effectively. My build is based on Armbian, though, rather than OpenWrt, purely because all the devices worked from the outset with Armbian, and I struggled for a long time to get OpenWrt working (and failed).

It can be a challenge to get OpenWrt working on some device but you can usually find pretty good support on IRC

One interesting thing, though, is that it doesn't seem necessary to patch the kernel to pass 802.1x traffic across the bridge. It just works, when I set bridge/group_forward_mask to 65528. I'm curious to know if I am missing a test case here, though?

It's OpenWrt specific, they have a patch that block it by default, but the author of the patch doesn't remember why it was needed, so we just added a sysctl to have the normal Linux behaviour

A cute trick that I use to prevent leaking of traffic onto the target network is to put the bridge and interfaces into a network namespace. Then only processes deliberately placed in the namespace are able to access the interface.

You should validate that with a "Throwing Star LAN Tap", as for example if you don't disable ipv6, it's doing DAD (duplicate address detection) when you ifup the interface or bridge, thus leaking some info.

I also have a python program watching tcpdump output for ARP, DNS, and UDP/TCP packets to update the arp tables, hosts files, and routing tables. It is not currently adding a default gateway if it identifies any non-RFC1918 traffic, but that would be a very simple thing to add. It also runs dnsmasq watching the directory that I write the DNS answers to, so that the user can refer to previously observed hosts by name, without ever putting their own DNS queries onto the wire.

That's nice! Just so you know the C part (doing all the detection) is not OpenWrt specific at all and can be reused easily, but it's not parsing DNS answers. With the OpenWrt integration we have 0 onsite configuration needed. We started with some bash parsing tcpdump output but it's too slow, it's command output parsing (never recommended), and you can't easily change the BPF filter (For performance reasons we try to not look at too much traffic). Also python is too heavy for small embedded OpenWrt routers and C for packet parsing work really well.

[RoganDawes]

It can be a challenge to get OpenWrt working on some device but you can usually find pretty good support on IRC

Yeah, I have had good experience working with the OpenWrt community.

One interesting thing, though, is that it doesn't seem necessary to patch the kernel to pass 802.1x traffic across the bridge. It just works, when I set bridge/group_forward_mask to 65528. I'm curious to know if I am missing a test case here, though?

It's OpenWrt specific, they have a patch that block it by default, but the author of the patch doesn't remember why it was needed, so we just added a sysctl to have the normal Linux behaviour

Ah, ok, makes sense.

A cute trick that I use to prevent leaking of traffic onto the target network is to put the bridge and interfaces into a network namespace. Then only processes deliberately placed in the namespace are able to access the interface.

You should validate that with a "Throwing Star LAN Tap", as for example if you don't disable ipv6, it's doing DAD (duplicate address detection) when you ifup the interface or bridge, thus leaking some info.

I do disable IPv6 on the 2 interfaces, as well as the bridge. It's cheating sure, but probably not missing too much at this point.

I also have a python program watching tcpdump output for ARP, DNS, and UDP/TCP packets to update the arp tables, hosts files, and routing tables. It is not currently adding a default gateway if it identifies any non-RFC1918 traffic, but that would be a very simple thing to add. It also runs dnsmasq watching the directory that I write the DNS answers to, so that the user can refer to previously observed hosts by name, without ever putting their own DNS queries onto the wire.

That's nice! Just so you know the C part (doing all the detection) is not OpenWrt specific at all and can be reused easily, but it's not parsing DNS answers.

I'm sure it could be reused, by someone who is familiar/competent with C :-D That someone is not me!!

With the OpenWrt integration we have 0 onsite configuration needed.

Yeah, same with my script too. I decided to back away from choosing an IP address on the switch side to use to target the victim PC, but that is not really needed. The bash script sets up the interfaces, the bridge and the network name space, and a couple of initial iptables rules to prevent packet leakage. Then it hands over to the python script that uses pypcap to get packet objects that it can parse.

We started with some bash parsing tcpdump output but it's too slow, it's command output parsing (never recommended), and you can't easily change the BPF filter (For performance reasons we try to not look at too much traffic). Also python is too heavy for small embedded OpenWrt routers and C for packet parsing work really well.

Absolutely understood. The R1S (and the AR750S for that matter) are a lot more competent devices, though. Which is not to say that having the implementation in C is a bad thing, but I prefer my script because it is hopefully a bit easier to understand from a teaching perspective.

[champtar]

A cute trick that I use to prevent leaking of traffic onto the target network is to put the bridge and interfaces into a network namespace. Then only processes deliberately placed in the namespace are able to access the interface.

You should validate that with a "Throwing Star LAN Tap", as for example if you don't disable ipv6, it's doing DAD (duplicate address detection) when you ifup the interface or bridge, thus leaking some info.

I do disable IPv6 on the 2 interfaces, as well as the bridge. It's cheating sure, but probably not missing too much at this point.

Still we also found some dumb typo in our script looking at the actual traffic the "Throwing Star", so I highly recommend validating with it

... I'm sure it could be reused, by someone who is familiar/competent with C :-D That someone is not me!!

Don't sell yourself short, I only do C when working on OpenWrt and each time I start I have to look again value/pointer/dereference :P

With the OpenWrt integration we have 0 onsite configuration needed.

Yeah, same with my script too. I decided to back away from choosing an IP address on the switch side to use to target the victim PC, but that is not really needed. The bash script sets up the interfaces, the bridge and the network name space, and a couple of initial iptables rules to prevent packet leakage. Then it hands over to the python script that uses pypcap to get packet objects that it can parse.

Ok so not using tcpdump but pycap ? Or both ?

We started with some bash parsing tcpdump output but it's too slow, it's command output parsing (never recommended), and you can't easily change the BPF filter (For performance reasons we try to not look at too much traffic). Also python is too heavy for small embedded OpenWrt routers and C for packet parsing work really well.

Absolutely understood. The R1S (and the AR750S for that matter) are a lot more competent devices, though. Which is not to say that having the implementation in C is a bad thing, but I prefer my script because it is hopefully a bit easier to understand from a teaching perspective.

So we have an AR750S but on OpenWrt it's using swconfig so only 1 interface visible to OpenWrt and you need to use some vlan tricks. With the migration to DSA and hardware nat/fast path we should be able to do the same job but at line rate in the future !!

[RoganDawes]

You should validate that with a "Throwing Star LAN Tap", as for example if you don't disable ipv6, it's doing DAD (duplicate address detection) when you ifup the interface or bridge, thus leaking some info. I do disable IPv6 on the 2 interfaces, as well as the bridge. It's cheating sure, but probably not missing too much at this point.

Still we also found some dumb typo in our script looking at the actual traffic the "Throwing Star", so I highly recommend validating with it

Fair enough! I am using my 750S as my upstream so I can see any unexpected packets that the gateway emits. I'm less worried about downstream, if they smell a rat, they will probably find it hiding behind the printer! :-)

I'm sure it could be reused, by someone who is familiar/competent with C :-D That someone is not me!!

Don't sell yourself short, I only do C when working on OpenWrt and each time I start I have to look again value/pointer/dereference :P

:-D

With the OpenWrt integration we have 0 onsite configuration needed.

Yeah, same with my script too. I decided to back away from choosing an IP address on the switch side to use to target the victim PC, but that is not really needed. The bash script sets up the interfaces, the bridge and the network name space, and a couple of initial iptables rules to prevent packet leakage. Then it hands over to the python script that uses pypcap to get packet objects that it can parse.

Ok so not using tcpdump but pycap ? Or both ?

tcpdump for the very first identification of the victim, primarily because of the -Q in flag, which can identify categorically the packets emitted by the victim. Pypcap after that.

We started with some bash parsing tcpdump output but it's too slow, it's command output parsing (never recommended), and you can't easily change the BPF filter (For performance reasons we try to not look at too much traffic). Also python is too heavy for small embedded OpenWrt routers and C for packet parsing work really well.

Absolutely understood. The R1S (and the AR750S for that matter) are a lot more competent devices, though. Which is not to say that having the implementation in C is a bad thing, but I prefer my script because it is hopefully a bit easier to understand from a teaching perspective.

So we have an AR750S but on OpenWrt it's using swconfig so only 1 interface visible to OpenWrt and you need to use some vlan tricks. With the migration to DSA and hardware nat/fast path we should be able to do the same job but at line rate in the future !!

Yeah, it will be great when the DSA migration is done. I really like the Slate! (https://twitter.com/RoganDawes/status/1203622183530123264)