Sporadic behavior despite active monitor mode

I have the TP-Link Archer T1U V1.0 card using the mt76 kernel module (which is supposed to possibly work as per https://github.com/seemoo-lab/opendrop/issues/24#issuecomment-537620580)
According to iw, Device supports active monitor (which will ACK incoming frames), so it should be good to go.
To begin, I killed wpa_supplicant, dhclient, NetworkManager, verified the process list, and don’t see anything else that might be otherwise interfering (as per https://github.com/seemoo-lab/opendrop/issues/21#issuecomment-532174115)
I set the regulatory domain to US (as per https://github.com/seemoo-lab/owl/issues/10#issuecomment-532151769)
I started owl with sudo owl -i wlx503eaa31c062 -v (which uses channel 6) and also tried channels 44 and 149. Owl puts the device in monitor mode.
There is no activity at all on channel 44, but channels 6 and 149 show activity.

When I turn on my iPhone (which appears to be c2:fa:2:66:0:6 here), I see messages like the following:

17:54:04 INFO : WLAN device: wlx503eaa31c062 (addr 50:3e:aa:31:c0:62)
17:54:04 INFO : Host device: awdl0
17:54:04 DEBUG: switch channel to 149 (slot 0)
17:54:08 DEBUG: peer c2:fa:2:66:0:6 () changed channel sequence to 149,149,149,149,0,0,4,4,6,149,149,149,0,0,4,4
17:54:08 INFO : add peer c2:fa:2:66:0:6 ()
17:54:08 DEBUG: new election tree: 50:3e:aa:31:c0:62 -> c2:fa:2:66:0:6 (met 61, ctr 3031)
17:54:10 DEBUG: peer c2:fa:2:66:0:6 () changed channel sequence to 149,149,149,0,0,0,4,4,6,149,149,0,0,0,4,4
17:54:10 DEBUG: peer ca:b:d0:c9:7d:53 () changed channel sequence to 149,149,149,149,149,149,40,40,6,149,149,149,149,149,40,40
17:54:10 INFO : add peer ca:b:d0:c9:7d:53 (3c50842e-9aef-4a36-9453-c0d8d88650c0)

My Apple devices appear correctly within AirDrop (and they are set to be discoverable by "Everyone"), but when I run sudo opendrop find -d it will only occasionally find a device, but not usually all devices, and it may take several minutes for any device to appear.

% sudo opendrop find -d
2020-10-27 17:56:11,410 INFO     opendrop.cli: Looking for receivers. Press enter to stop ...
2020-10-27 17:56:31,514 DEBUG    opendrop.client: Add service 0689a66e61bf._airdrop._tcp.local.
2020-10-27 17:56:31,514 DEBUG    opendrop.cli: AirDrop service found: a043b3ce-ee6e-4d40-bcaa-6ce4176845c6.local., fe80::18cc:c0ff:fecf:40e1:8770, ID 0689a66e61bf
2020-10-27 17:56:31,514 DEBUG    opendrop.client: Send /Discover request
/usr/local/lib/python3.6/dist-packages/zeroconf/__init__.py:1541: FutureWarning: <opendrop.client.AirDropBrowser object at 0x7f2a08b0e400> has no update_service method. Provide one (it can be empty if you don't care about the updates), it'll become mandatory.
FutureWarning,
2020-10-27 17:56:31,665 DEBUG    opendrop.client: /Discover request successful
2020-10-27 17:56:31,666 INFO     opendrop.cli: Found  index 0  ID 0689a66e61bf  name tom’s MacBook Pro

When a device is discovered, I can usually send a file over with opendrop send (yay).

When I put opendrop in receive mode, it almost never shows up on my iPhone or Mac. I got it to show up on rare occasion after waiting 10 to 15 minutes and managed to do one receive (another one failed).

% sudo opendrop -d receive -d
2020-10-27 17:58:23,292 INFO     opendrop.server: Announcing service: host nuc, address fe80::523e:aaff:fe31:c062, port 8771
2020-10-27 17:58:24,093 INFO     opendrop.server: Starting HTTPS server

In the rare occasions I'm able to receive a file in opendrop, I get a many messages like 22:00:23 ERROR: unable to inject packet (send: Resource temporarily unavailable) during the transfer in OWL.

I opened this issue in the owl repository since I believe that the issue lies in the owl implementation and not in the opendrop tool itself, but I can't confirm. What can I do to get this to work or debug this further?

I was able to maybe narrow the problem down a bit.

I have a MacBook running macOS Catalina 10.15.7 (fe80::fc77:26ff:febc:5a5d), iPhone running iOS 14.0 (fe80::1c0d:aaff:fe4d:5a79) and a Linux box running owl (fe80::523e:aaff:fe31:c062) on the AWDL network. ff02::fb is the broadcast address. The iPhone has a share sheet open and the Mac has a Finder Airdrop window open.

From the MacBook, I can ping both the iPhone and Linux box directly without any issues (latency is between 4ms and 400ms). When I ping the broadcast, I get duplicates back from both the Linux box and the iPhone:

% ping6 ff02::fb%awdl0
PING6(56=40+8+8 bytes) fe80::fc77:26ff:febc:5a5d%awdl0 --> ff02::fb%awdl0
ping6: failed to get receiving hop limit
16 bytes from fe80::fc77:26ff:febc:5a5d%awdl0, icmp_seq=0 hlim=64 time=0.219 ms
16 bytes from fe80::1c0d:aaff:fe4d:5a79%awdl0, icmp_seq=0 hlim=64 time=531.305 ms
16 bytes from fe80::523e:aaff:fe31:c062%awdl0, icmp_seq=0 hlim=64 time=532.701 ms
16 bytes from fe80::523e:aaff:fe31:c062%awdl0, icmp_seq=0 hlim=64 time=925.133 ms
16 bytes from fe80::1c0d:aaff:fe4d:5a79%awdl0, icmp_seq=0 hlim=64 time=926.269 ms
16 bytes from fe80::fc77:26ff:febc:5a5d%awdl0, icmp_seq=1 hlim=64 time=0.228 ms
16 bytes from fe80::1c0d:aaff:fe4d:5a79%awdl0, icmp_seq=1 hlim=64 time=446.991 ms
16 bytes from fe80::1c0d:aaff:fe4d:5a79%awdl0, icmp_seq=1 hlim=64 time=575.425 ms
16 bytes from fe80::523e:aaff:fe31:c062%awdl0, icmp_seq=1 hlim=64 time=576.498 ms
16 bytes from fe80::523e:aaff:fe31:c062%awdl0, icmp_seq=1 hlim=64 time=968.384 ms
16 bytes from fe80::1c0d:aaff:fe4d:5a79%awdl0, icmp_seq=1 hlim=64 time=968.629 ms
...

From the Linux box running owl I can ping the Mac, however I'm having trouble pinging the iPhone. Sometimes none of the pings go through, sometimes just the first one (icmp_seq=1). Is this a driver issue with the mt76 or a bug in owl? What would be special about the first ping compared to the others?

PING fe80::1c0d:aaff:fe4d:5a79%awdl0(fe80::1c0d:aaff:fe4d:5a79%awdl0) 56 data bytes
64 bytes from fe80::1c0d:aaff:fe4d:5a79%awdl0: icmp_seq=1 ttl=64 time=163 ms
^C
--- fe80::1c0d:aaff:fe4d:5a79%awdl0 ping statistics ---
219 packets transmitted, 1 received, 99% packet loss, time 223191ms
rtt min/avg/max/mdev = 163.962/163.962/163.962/0.000 ms

When I ping the broadcast from the Linux box, I get delayed replies (up to ~15s) from the Mac, but none from the iPhone.

% ping ff02::fb%awdl0
PING ff02::fb%awdl0(ff02::fb%awdl0) 56 data bytes
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=1 ttl=64 time=0.030 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=3 ttl=64 time=0.043 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=4 ttl=64 time=0.043 ms
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=1 ttl=64 time=3598 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=2 ttl=64 time=2595 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=3 ttl=64 time=1572 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=4 ttl=64 time=916 ms (DUP!)
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=5 ttl=64 time=0.047 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=6 ttl=64 time=0.046 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=7 ttl=64 time=0.046 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=8 ttl=64 time=0.045 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=9 ttl=64 time=0.051 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=10 ttl=64 time=0.047 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=11 ttl=64 time=0.048 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=12 ttl=64 time=0.045 ms
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=5 ttl=64 time=7268 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=10 ttl=64 time=2155 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=11 ttl=64 time=1131 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=12 ttl=64 time=107 ms (DUP!)
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=13 ttl=64 time=0.047 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=14 ttl=64 time=0.047 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=15 ttl=64 time=0.045 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=16 ttl=64 time=0.047 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=17 ttl=64 time=0.041 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=18 ttl=64 time=0.045 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=19 ttl=64 time=0.044 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=20 ttl=64 time=0.045 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=21 ttl=64 time=0.046 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=22 ttl=64 time=0.045 ms
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=13 ttl=64 time=9586 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=14 ttl=64 time=8571 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=15 ttl=64 time=7555 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=16 ttl=64 time=6531 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=17 ttl=64 time=5508 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=18 ttl=64 time=4484 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=19 ttl=64 time=3460 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=20 ttl=64 time=2436 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=21 ttl=64 time=1412 ms (DUP!)
64 bytes from fe80::fc77:26ff:febc:5a5d%awdl0: icmp_seq=22 ttl=64 time=395 ms (DUP!)
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=23 ttl=64 time=0.049 ms
64 bytes from fe80::523e:aaff:fe31:c062%awdl0: icmp_seq=24 ttl=64 time=0.048 ms

Note that this is on channel 149 with the regulatory domain set to US. If the regulatory domain isn't set, none of the packets go out (consistent with https://github.com/seemoo-lab/owl/issues/10#issuecomment-532151769).

Maybe @Kulawik @neilalexander have any insights on this since it appears you have the same card?

@thomasst which part of your issue was resolved with #39?

@schmittner Thanks for checking in. #39 fixed queueing up the packets and responding in bulk in the scenario where I ping the broadcast. In my last comment above you can see that fe80::fc77:26ff:febc:5a5d%awdl0 was responding in batches to the pings, since they were previously queued up in the circular buffer -- this is now fixed.

However, there is still packet when sending packets via owl, on both TP-Link Archer T1U V1.0 as well as the Alfa AWUS036NHA. Overall, the Alfa card is performing better than the TP-Link. I suspect it might have to do with the timing of when the packets are sent out, but I'm not familiar enough with AWDL to figure it out. On the machine running owl, I can see all packets going out in wireshark on both awdl0 as well as the physical wi-fi interface, but I can't see it in Wireshark coming in on the Mac, so it appears the packet somehow gets lost "in the air".

With the Alfa card, there is frequent and sporadic packet loss without a specific pattern that I could figure out. On the Archer, certain devices, like my iPhone, barely receive any packets. The issue described above ("Sometimes none of the pings go through, sometimes just the first one (icmp_seq=1).") is still happening.

Do you have any ideas why this might be? If you have time to help out, I'm happy to do any more debugging and provide more details. Thanks!

I see, thanks for clarifying. Can you assert that packets actually arrive at the Mac and not get lost over the air? To do this, set your Mac to monitor mode and record packets on channel 149 (in your case).

I can readily reproduce this here. I have an iPhone running iOS 15.1 and a MacBook Pro running Catalina 10.15.7 with OWL on a Raspberry Pi running kernel 5.10. I'm using a TP-Link Archer T1U V1.0 using the mt76 driver. MBP can send and receive to OWL just fine but the iPhone shows the sporadic traffic patterns described by @thomasst.

Running tcpdump on the Pi, while executing a ping to the link-local address of the iPhone, ICMPv6 echo request packets are being sent over the awdl0 interface and physical WLAN interfaces. Monitoring the rvi0 interface (which is a virtual interface representing all traffic sent to/from the iPhone, exposed on the MacBook as described here) shows that the iPhone is happily responding with an ICMPv6 echo reply, but those are only received sporadically by the Pi - tcpdump on either the awdl0 or physical interface shows that the majority of replies are somehow lost.

When monitoring awdl0 on the Pi I can see other traffic (e.g. mDNS multicast traffic) being received from the iPhone but any unicast traffic seems to be sporadic. When I run tcpdump on the physical WLAN interface while the ping is running I can see the occasional sporadic ping reply being received and plenty of radiotap traffic being received from the iPhone, as well as multicast packets. That seems to indicate that the IPv6 ping reply sent from the iPhone is dropped somewhere along the way.

When pinging the iPhone's link-local address via my MacBook's awdl0 interface then it pings consistently.

Additionally all of this just works if I do the same things from the MBP (rather than the iPhone).

I'm unsure how to go about debugging this further - any pointers really appreciated! I'm really confused as to why the MBP works fine but the iPhone does not.

seemoo-lab / owl

Sporadic behavior despite active monitor mode #38