morrownr / USB-WiFi

USB WiFi Adapter Information for Linux
2.41k stars 161 forks source link

List of Bug Reports for the mt7921au chipset / mt7921u driver... #107

Open morrownr opened 1 year ago

morrownr commented 1 year ago

This issue is for maintaining a list of problematic issues that need work. This list will be maintained and updated in this first post by @morrownr . Please add posts to this issue as you have updated information for the existing BUGs in the list or if you have information about a new BUG. Thank you.

Dear Mediatek devs... help is appreciated.


Bug: (2024-04-18) See: https://github.com/morrownr/USB-WiFi/issues/392 . WDS/4addr not supported in AP mode. First reported with Alfa AXML adapter that uses the mt7921au chipset and mt7921u driver). The OP is unable to use WDS/4addr in AP mode.

Status: Open

Info: It was reported that this capability does work with an adapter that uses the mt7612u chipset/driver.


Bug: (2024-03-26) See: https://github.com/morrownr/USB-WiFi/issues/378 Wifi adapter not showing up. First reported with Alfa AXML adapter that uses the mt7921au chipset and mt7921u driver). The adapter is non-functional until using the workaround below.

Status: Open

Workaround: the workaround is to run modprobe -r btusb first, then plug in the usb wifi adapter.

More input is needed. Is this a problem with btusb?


Bug: (2023-12-22) Many Linux distros are detecting Bluetooth capability in mt7921au based adapters but none of the adapters on the market have Bluetooth turned on so it won't work. Linux should not be detecting Bluetooth capability when it is actually not available.

Status: Open and ongoing

Here is a link to a location where you can get a copy of the Intel White Paper that explains the details of why USB3 capable WiFi adapters should not have Bluetooth capability turned on:

https://www.usb.org/document-library/usb-30-radio-frequency-interference-impact-24-ghz-wireless-devices

USB3 WiFi adapters should not have Bluetooth turned on as the USB3 will cause interference with Bluetooth. If makers decide they really want Bluetooth capability in an adapter then they need to limit wifi to USB2 capability. All adapters with the mt7921au chipset that I am aware of have Bluetooth turned off so WiFi can operate in USB3 mode. However, there is a bug in that Bluetooth capability is still being detected by Linux distros and the driver/firmware is loading. Systems act like Bluetooth is available but when you try to use the Bluetooth, it won't work. It is not clear to me how this can be fixed but it really does need to be fixed.

This is not a problem with PCIe cards. I have a mt7922 based PCIe card. Wifi and Bluetooth work well together because wifi uses the PCIe bus and not USB. Please understand that issue in this bug is not exclusive to this chipset. This is an issue will all USB WiFi adapters. The adapters that have USB wifi capability and BT capabilities over the years have limited USB to USB2 to avoid the problem of interference.


Bug: (2023-12-07) Active monitor mode breaks driver.

Status: open

Reporter: @ZerBea Link: https://github.com/openwrt/mt76/issues/839 Problem: Using Active Monitor mode breaks the driver

Driver reports that active monitor mode is possible:

$ iw list | grep active Device supports active monitor (which will ACK incoming frames)

But if hcxdumptool set active monitor mode, it stops working.

If active monitor mode is disabled, everything's fine

0 ERROR(s) during runtime 638 Packet(s) captured by kernel 0 Packet(s) dropped by kernel 1 SHB written to pcapng dumpfile 1 IDB written to pcapng dumpfile 1 ECB written to pcapng dumpfile 83 EPB written to pcapng dumpfile

exit on sigterm I don't think the problem is related to hcxdumptool, because it can be reproduced with iw, ip link and tshark, too:

$ sudo ip link set wlp22s0f0u4i3 down $ sudo iw dev wlp22s0f0u4i3 set type monitor $ sudo ip link set wlp22s0f0u4i3 up $ tsahrk -i wlp22s0f0u4i3 22 packets captured

$ sudo ip link set wlp22s0f0u4i3 down $ sudo iw dev wlp22s0f0u4i3 set monitor active $ sudo ip link set wlp22s0f0u4i3 up $ tshark -i wlp22s0f0u4i3 Capturing on 'wlp22s0f0u4i3' ^C 0 packets captured

Background: Running active monitor mode, the device ACK incoming frames addressed to the virtual MAC of the device. This feature is really useful to perform PMKID attacks. At the moment, active monitor mode is working on:

mt76x0u mt76x2u

It is not working on:

mt7601u mt7921u

I see two options: active monitor mode should be fixed, or active monitor mode capability should not be reported by the driver

mt7601u $ iw list | grep active Device supports active monitor (which will ACK incoming frames)

mt7921u $ iw list | grep active Device supports active monitor (which will ACK incoming frames)


Bug: LED does not function in several of the usb wifi adapters that use the mt7921au chipset.

Status: open, it is unclear what the problem is.

Reported by @morrownr Confirmed by numerous users.


Bug: AP Mode DFS (5 GHz) support is non-functional Status: open

Reported by @morrownr Confirmed by numerous users.

This is really a serious omission in that in many places in the world there are limited non-DFS channels available leading to high levels of congestion.

Dear Mediatek, does your usb chipset competitor support DFS channels in AP Mode? Yes they do. See: out-of-kernel drivers for rtl8812au, rtl8811au, rtl8812bu and rtl8811cu. You need to think about this. Sincerely.


Bug: txpower reading is showing as unusually low as in 3 dBm using iw. Status: open

Reported by several individuals.

This reading must be wrong because actual usage suggests the reading should be much higher.


Bug: (feature request) mt7921u driver does not support 2 interfaces of AP mode on one adapter Status: open

Reported by @whitslack

mt7921u driver does not support 2 instances of AP mode whereas this was common on some drivers for older adapters.

Now:

valid interface combinations:

     * #{ managed, P2P-client } <= 2, #{ AP, P2P-GO } <= 1,
       total <= 2, #channels <= 2

What we want:

valid interface combinations:

     * #{ managed, P2P-client } <= 2, #{ AP, P2P-GO } <= 2,
       total <= 2, #channels <= 2

Bug: connection is dropped and the only way to correct the situation is to reboot (AP mode) Status: open

Testing to see if SG helps performance:

scatter-gather test with mt7921au based adapter

Issue: connection drops and the only resolution is to reboot the system.

Raspberry Pi 4B RasPiOS 2023-05-03

I changed the modulate parameter and rebooted between each test so as to alternate on and off.

iperf3 -c 192.168.1.1 -t 300

scatter-gather off (disable_usb_sg=1)

1:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec  19.9 GBytes   569 Mbits/sec    4             sender
[  5]   0.00-300.01 sec  19.9 GBytes   569 Mbits/sec                  receiver

2: 
[  5]   0.00-300.00 sec  19.9 GBytes   570 Mbits/sec    5             sender
[  5]   0.00-300.01 sec  19.9 GBytes   570 Mbits/sec                  receiver

3:
[  5]   0.00-300.00 sec  20.0 GBytes   573 Mbits/sec    2             sender
[  5]   0.00-300.01 sec  20.0 GBytes   573 Mbits/sec                  receiver

scatter-gather on (disable_usb_sg=0)

1:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec  19.9 GBytes   570 Mbits/sec    1             sender
[  5]   0.00-300.01 sec  19.9 GBytes   570 Mbits/sec                  receiver

2:
[  5]   0.00-300.00 sec  20.0 GBytes   572 Mbits/sec   48             sender
[  5]   0.00-300.01 sec  20.0 GBytes   572 Mbits/sec                  receiver

3.
[  5]   0.00-300.00 sec  19.9 GBytes   571 Mbits/sec    0             sender
[  5]   0.00-300.02 sec  19.9 GBytes   571 Mbits/sec                  receiver

Observation: So much for needing to average the results. I was careful to check that sg was on or off. I have no explanation for how the results could be so close. I see no evidence that sg is providing any performance increase.

Previous to this testing session, I have been able to see the issue of the connection being dropped and only a reboot will connect the situation. It happened twice a few days ago while testing with sg on. There is a history of this with mt7612u adapters. I have yet to duplicate the issue with sg off.

Conclusion: Further testing on different platforms is needed. I will test x86_64 next. Given the history of sg causing problems such as connections dropping that can only be corrected with a reboot, it may be better for the default to be disable_usb_sg=1 with a follow up to determine what the problem is.


morrownr commented 1 year ago

My mind has turned to mush as I get older. I guess I jusr automatically upgrade it and forget I did so. Time to file a bug with the RasPi folks.

gifter77 commented 1 year ago

I cannot seem to reproduce on this fresh install though, even with this version of the firmware. This is very odd.

EasyNetDev commented 1 year ago

OK. My setup is a RPI 3 with kernel 6.1.21+. I've addeed the latest version of firmware for MT7921u I could found:

[280849.036366] usbcore: registered new interface driver mt7921u
[280849.046172] mt7921u 1-1.4:1.3: HW/SW Version: 0x8a108a10, Build Time: 20220908210919a
[280849.371419] mt7921u 1-1.4:1.3: WM Firmware Version: ____010000, Build Time: 20220908211021
[292744.962519] usbcore: deregistering interface driver mt7921u
[292822.280258] usbcore: registered new interface driver mt7921u
[292822.283238] mt7921u 1-1.4:1.3: HW/SW Version: 0x8a108a10, Build Time: 20230331110902a
[292822.299917] mt7921u 1-1.4:1.3: WM Firmware Version: ____010000, Build Time: 20230331110939

And here are details of the driver:

filename:       /lib/modules/6.1.21-v8+/kernel/drivers/net/wireless/mediatek/mt76/mt7921/mt7921u.ko.xz
license:        Dual BSD/GPL
author:         Lorenzo Bianconi <lorenzo@kernel.org>
firmware:       mediatek/WIFI_MT7961_patch_mcu_1_2_hdr.bin
firmware:       mediatek/WIFI_RAM_CODE_MT7961_1.bin
srcversion:     774ADC863988DEEFA8EF149
alias:          usb:v0E8Dp7961d*dc*dsc*dp*icFFiscFFipFFin*
depends:        mt76-connac-lib,mt76-usb,mt7921-common,mt76
intree:         Y
name:           mt7921u
vermagic:       6.1.21-v8+ SMP preempt mod_unload modversions aarch64
bjlockie commented 1 year ago

You run iperf3 -s on the pi and iperf3 -3 [ip of pi] to test?

gifter77 commented 1 year ago

Oh I got a new one while trying to reproduce the other one:

[ 1178.838761] mt7921u 1-1.3:1.0: vendor request req:63 off:8154 failed:-71
[ 1178.838816] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.839302] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.839562] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.839826] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.840088] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.840350] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.840614] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.840876] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.841138] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.841428] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.841689] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.841952] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.842214] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.842472] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.842762] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.843025] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.843312] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.843570] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.843833] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.844094] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.844357] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.844621] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.844881] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.845157] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.845432] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.845694] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.845955] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.846224] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.846478] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.846788] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.847113] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.847466] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.847813] usb 1-1.3: USB disconnect, device number 5
[ 1178.847834] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.848111] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.848369] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.848960] mt7921u 1-1.3:1.0: tx urb failed: -71
[ 1178.867031] wlan0: deauthenticating from <redacted> by local choice (Reason: 3=DEAUTH_LEAVING)
[ 1179.382976] wlan0: failed to remove key (0, <redacted>) from hardware (-19)
[ 1179.388604] ------------[ cut here ]------------
[ 1179.388617] WARNING: CPU: 0 PID: 1953 at net/wireless/util.c:1388 cfg80211_calculate_bitrate_he+0x260/0x2b8 [cfg80211]
[ 1179.388761] Modules linked in: xt_DSCP xt_tcpudp nft_compat nf_tables nfnetlink ctr aes_arm64 aes_generic libaes ccm 8021q garp stp llc mt7921u mt7921_common mt76_connac_lib mt76_usb mt76 mac80211 imx708 dw9807_vcm libarc4 spidev cfg80211 rfkill vc4 snd_soc_hdmi_codec drm_display_helper cec drm_dma_helper drm_kms_helper bcm2835_codec(C) rpivid_hevc(C) bcm2835_unicam bcm2835_v4l2(C) snd_soc_core v4l2_dv_timings bcm2835_isp(C) v4l2_mem2mem bcm2835_mmal_vchiq(C) v4l2_fwnode snd_compress v4l2_async videobuf2_dma_contig videobuf2_vmalloc snd_pcm_dmaengine snd_bcm2835(C) videobuf2_memops videobuf2_v4l2 v3d snd_pcm gpu_sched videobuf2_common raspberrypi_hwmon i2c_mux_pinctrl drm_shmem_helper i2c_mux snd_timer videodev i2c_brcmstb snd spi_bcm2835 i2c_bcm2835 mc vc_sm_cma(C) syscopyarea sysfillrect sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio i2c_dev drm fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[ 1179.388937] CPU: 0 PID: 1953 Comm: kworker/0:1 Tainted: G         C         6.1.21-v8+ #1642
[ 1179.388944] Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
[ 1179.388949] Workqueue: usb_hub_wq hub_event
[ 1179.388965] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1179.388971] pc : cfg80211_calculate_bitrate_he+0x260/0x2b8 [cfg80211]
[ 1179.389047] lr : 0x1856ac15
[ 1179.389052] sp : ffffffc00953afc0
[ 1179.389055] x29: ffffffc00953afc0 x28: ffffff804f00db72 x27: ffffff8049600000
[ 1179.389065] x26: ffffff80496003a0 x25: ffffff8049697030 x24: ffffff8049697000
[ 1179.389074] x23: ffffff80449f6900 x22: 00000000000000d0 x21: ffffff80496970d0
[ 1179.389083] x20: ffffff804f68ea38 x19: ffffffefaaaad858 x18: 0cea122a0dac8927
[ 1179.389092] x17: 000000000b9f76c0 x16: 0675091506d65a47 x15: 0000000005cfbb60
[ 1179.389101] x14: 0240717102625a00 x13: 000000000206cc80 x12: 010f4471011f2ba0
[ 1179.389110] x11: 0000000000f42400 x10: 0000000000000004 x9 : 000063ff00008556
[ 1179.389119] x8 : 0000c80100018fff x7 : 000027ff00002c71 x6 : 00003201000042ab
[ 1179.389127] x5 : 0000180000001aac x4 : 0000000000000003 x3 : 0087a238008f9a27
[ 1179.389136] x2 : 00000000007a1200 x1 : 000014000000163a x0 : ffffff804f68ea38
[ 1179.389145] Call trace:
[ 1179.389148]  cfg80211_calculate_bitrate_he+0x260/0x2b8 [cfg80211]
[ 1179.389223]  cfg80211_calculate_bitrate+0x224/0x5d0 [cfg80211]
[ 1179.389297]  nl80211_put_sta_rate+0x6c/0x3d8 [cfg80211]
[ 1179.389370]  nl80211_send_station.isra.99+0x6e4/0xc00 [cfg80211]
[ 1179.389444]  cfg80211_del_sta_sinfo+0xbc/0x1c8 [cfg80211]
[ 1179.389517]  __sta_info_destroy_part2+0xd0/0x158 [mac80211]
[ 1179.389656]  __sta_info_flush+0x124/0x190 [mac80211]
[ 1179.389737]  ieee80211_set_disassoc+0x12c/0x600 [mac80211]
[ 1179.389818]  ieee80211_mgd_deauth+0x164/0x308 [mac80211]
[ 1179.389897]  ieee80211_deauth+0x20/0x30 [mac80211]
[ 1179.389977]  cfg80211_mlme_deauth+0xb4/0x218 [cfg80211]
[ 1179.390056]  cfg80211_mlme_down+0x8c/0x98 [cfg80211]
[ 1179.390130]  cfg80211_disconnect+0x174/0x210 [cfg80211]
[ 1179.390203]  __cfg80211_leave+0x190/0x1a8 [cfg80211]
[ 1179.390277]  cfg80211_leave+0x3c/0x58 [cfg80211]
[ 1179.390350]  cfg80211_netdev_notifier_call+0xdc/0x540 [cfg80211]
[ 1179.390423]  raw_notifier_call_chain+0x5c/0x80
[ 1179.390433]  call_netdevice_notifiers_info+0x60/0xb0
[ 1179.390440]  __dev_close_many+0x68/0x158
[ 1179.390445]  dev_close_many+0x94/0x130
[ 1179.390450]  dev_close.part.117+0x50/0x80
[ 1179.390455]  dev_close+0x24/0x30
[ 1179.390461]  cfg80211_shutdown_all_interfaces+0x4c/0x108 [cfg80211]
[ 1179.390534]  ieee80211_remove_interfaces+0x5c/0x238 [mac80211]
[ 1179.390618]  ieee80211_unregister_hw+0x50/0x100 [mac80211]
[ 1179.390698]  mt76_unregister_device+0x48/0x68 [mt76]
[ 1179.390729]  mt7921u_disconnect+0x4c/0xc0 [mt7921u]
[ 1179.390740]  usb_unbind_interface+0x78/0x228
[ 1179.390749]  device_remove+0x78/0x88
[ 1179.390756]  device_release_driver_internal+0xe8/0x190
[ 1179.390762]  device_release_driver+0x20/0x30
[ 1179.390767]  bus_remove_device+0x130/0x140
[ 1179.390772]  device_del+0x184/0x3b8
[ 1179.390778]  usb_disable_device+0x94/0x160
[ 1179.390783]  usb_disconnect+0xc4/0x2c0
[ 1179.390791]  hub_event+0x6e0/0x1300
[ 1179.390797]  process_one_work+0x208/0x480
[ 1179.390805]  worker_thread+0x50/0x428
[ 1179.390811]  kthread+0xfc/0x110
[ 1179.390817]  ret_from_fork+0x10/0x20
[ 1179.390824] ---[ end trace 0000000000000000 ]---
whitslack commented 1 year ago

usb 1-1.3: USB disconnect, device number 5

That looks more like a USB power issue or a loose connector, broken cable, etc. It's not surprising that the driver will come to a very noisy halt when the device is effectively unplugged while in use.

gifter77 commented 1 year ago

That looks more like a USB power issue or a loose connector, broken cable, etc. It's not surprising that the driver will come to a very noisy halt when the device is effectively unplugged while in use.

Ok then that is a side effect of me unplugging and replugging the adapter when I lost the connection.

gifter77 commented 1 year ago

@morrownr so it seems that the way to reproduce the issue on a fresh install is to get the Pi to download stuff using iperf and also starts some webrtc service on the pi at the same time to stream the Pi camera output. Then when viewing this stream from another computer, it triggers the wifi driver crash. This is extremely strange. Maybe the reproduction scenario could be simplified to downloading stuff on the Pi and sending out UDP multicast packets at the same time.

morrownr commented 1 year ago

also starts some webrtc service on the pi at the same time to stream the Pi camera output.

Then when viewing this stream from another computer, it triggers the wifi driver crash. This is extremely strange.

Agree. That is why I was being so picky about getting everything up to the latest version on a fresh configuration. We don't stand a chance of providing a good bug report without having it narrowed down to something simply and repeatable.

You had not previously mentioned camera output. I am following along trying to duplicate the issue. What camera and app are you using? I have a webcam and can compile a streamer app but I need to see if I can get as close as possible to your setup.

Can you also make me a list of everything that is plugged into a usb port? And keep it updated as we go along. I have to see what you are doing to be able to try to duplicate the problem.

Devs have their own techniques for many things. When it comes to Pi's and usb, I am reluctant to just start digging in code as that has proved to be futile unless first I have a very simple, easy to reproduce scenario.

gifter77 commented 1 year ago

You had not previously mentioned camera output. I am following along trying to duplicate the issue. What camera and app are you using? I have a webcam and can compile a streamer app but I need to see if I can get as close as possible to your setup.

I'm using the pi camera v3 so it is not plugged on a USB port but on the CSI port of the Pi. The WebRTC streamer is this one: https://github.com/ayufan/camera-streamer

Can you also make me a list of everything that is plugged into a usb port? And keep it updated as we go along. I have to see what you are doing to be able to try to duplicate the problem.

It can be reproduced with only the adapter plugged on a USB2 or USB3 port.

I doubt this is specific to using a camera and streaming some WebRTC packets though. Maybe trying to send any UDP multicast packet from the Pi would work. Remember the original reporter of this bug in this thread (@leezu) could trigger it with some heavy load from a client connected to his AP.

morrownr commented 1 year ago

@gifter77

Status report:

[  5]   0.00-1.00   sec  65.3 MBytes   548 Mbits/sec    0   1.86 MBytes       
[  5]   1.00-2.00   sec  72.5 MBytes   608 Mbits/sec    0   1.86 MBytes       
[  5]   2.00-3.00   sec  75.0 MBytes   629 Mbits/sec    0   1.86 MBytes       
[  5]   3.00-4.00   sec  72.5 MBytes   608 Mbits/sec    0   1.86 MBytes       
[  5]   4.00-5.00   sec  73.8 MBytes   618 Mbits/sec    0   1.86 MBytes       
[  5]   5.00-6.00   sec  73.8 MBytes   619 Mbits/sec    0   1.86 MBytes       
[  5]   6.00-7.00   sec  72.5 MBytes   608 Mbits/sec    0   1.86 MBytes       
[  5]   7.00-8.00   sec  75.0 MBytes   629 Mbits/sec    0   1.86 MBytes       
[  5]   8.00-9.00   sec  73.8 MBytes   619 Mbits/sec    0   1.86 MBytes       
[  5]   9.00-10.00  sec  71.2 MBytes   598 Mbits/sec    0   1.86 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   725 MBytes   608 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   724 MBytes   607 Mbits/sec                  receiver

The above uses a RasPi4B v1.2 with a lone CF-951AX in managed mode connected to my main wifi router.

The wifi router runs OpenWRT 22.03.5. Channel 100 DFS. No other AP's on the channel.

RasPiOS is 64 bit 2023-05-03. I have not upgrade the old firmware for the mt7921u driver but will. This takes a lot of time and I have other things going on.

The only thing in a port is the wifi adapter in a USB3 port. I have confirmed that it is running in USB3 mode but that was a waste of time as USB2 cannot do the speeds we see above.

The only ugliness I am seeing in dmesg is:

[ 10.943065] Bluetooth: hci0: Opcode 0x c03 failed: -110 [ 12.959031] Bluetooth: hci0: Opcode 0x c03 failed: -110

That is caused by RasPiOS detecting the bt support in the cf-951ax, even though Comfast turned the support off. I can get rid of this by deleting the bt firmware. I'll do that when I upgrade the firmware for the wifi.

Do you want me to do something like?

iperf3 -c 192.168.1.1 -t 600

...to see if heating things up will show something?

Of note: The speeds you see above would be higher, probably around 650 Mbps, but I am running headless so that takes some of the bandwidth.

What a thread! And we don't seem to be any closer to narrowing down the issue.

@morrownr

gifter77 commented 1 year ago

I think what you could try is getting the Pi much further from your router before running the test. Mine is pretty far and the signal level is like -70dB. So I'm not getting more than 100Mb/s.

morrownr commented 1 year ago

I think what you could try is getting the Pi much further from your router

I moved the Pi. It was about 30 feet from the router (10 meters) but was line of sight. Now the distance is about 50 feet (16 meters or so) and has 3 walls in the way.

wavemon went from showing -38 to -52.

I ran...

iperf3 -c 192.168.1.1 -t 300

Average 505 Mbps. Speed was consistent. Nothing in the log other than the normal crap related to Broadcom's crap Linux support.

I can jack up the overclocking and run iperf3 longer?

morrownr commented 1 year ago

So, time for a little abuse to see if I can trigger something.

I updated the wifi firmware and deleted the bt firmware.

over_voltage=6
arm_freq=2000
gpu_freq=750

Result:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec  17.8 GBytes   509 Mbits/sec    2             sender
[  5]   0.00-300.01 sec  17.8 GBytes   508 Mbits/sec                  receiver

Still at about 16 meters and 3 walls.

More abuse...

The Pi just dropped out on this attempt. Since I was running headless, now I need to find a solution to seeing what is in the log.

gifter77 commented 1 year ago

The Pi just dropped out on this attempt. Since I was running headless, now I need to find a solution to seeing what is in the log.

You can plug an ethernet cable and check that way I guess. That's crazy you are getting these kind of speeds 16 meters away. I get 100 Mbps like less than 10 meters away, though there is a ceiling and a concrete wall in between my router and my Pi in the garage.

gifter77 commented 1 year ago

Another way to get the connection back up is to unplug and replug the adapter. I'm guessing you have an udev rule for it.

morrownr commented 1 year ago

Another way to get the connection back up is to unplug and replug the adapter.

Ah, so now I need to trigger the problem again and see if this works. I do have ethernet cables but I'd prefer this easier solutions work.

My router is Wave 2 capable and it appears the mt7921au chipset is Wave 2 capable as well. My router is also capable of DFS channels and it is on channel 100 (5 GHz) right now. Totally clear air. No other AP's on the channel at all. The router also does 160 MHz channel width and I have it set just because I can but I have no client devices that can do 160 MHz channel width.

I have a speed test up:

https://github.com/morrownr/USB-WiFi/blob/main/home/Speed_Comparison_Test.md

Now, let me see if I can trigger the drop out again and get a log.

morrownr commented 1 year ago

Having trouble triggering the problem again:

$ iperf3 -c 192.168.1.1 -t 600
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.119 port 32992 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  52.6 MBytes   441 Mbits/sec    0   1.90 MBytes       
[  5]   1.00-2.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   2.00-3.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   3.00-4.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   4.00-5.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   5.00-6.00   sec  55.0 MBytes   461 Mbits/sec    0   1.90 MBytes       
[  5]   6.00-7.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   7.00-8.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   8.00-9.00   sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]   9.00-10.00  sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]  10.00-11.00  sec  55.0 MBytes   461 Mbits/sec    0   1.90 MBytes       
[  5]  11.00-12.00  sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]  12.00-13.00  sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]  13.00-14.00  sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]  14.00-15.00  sec  57.5 MBytes   482 Mbits/sec    0   1.90 MBytes       
[  5]  15.00-16.00  sec  55.0 MBytes   462 Mbits/sec    0   1.90 MBytes       
[  5]  16.00-17.00  sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]  17.00-18.00  sec  56.2 MBytes   471 Mbits/sec    0   1.90 MBytes       
[  5]  18.00-19.00  sec  32.5 MBytes   273 Mbits/sec    0   1.90 MBytes       
[  5]  19.00-20.00  sec  30.0 MBytes   252 Mbits/sec    0   1.90 MBytes       
[  5]  20.00-21.00  sec  31.2 MBytes   262 Mbits/sec    0   1.90 MBytes       
[  5]  21.00-22.00  sec  26.2 MBytes   220 Mbits/sec    0   1.90 MBytes       
[  5]  22.00-23.00  sec  26.2 MBytes   220 Mbits/sec    0   1.90 MBytes       
[  5]  23.00-24.00  sec  26.2 MBytes   220 Mbits/sec    0   1.90 MBytes       
[  5]  24.00-25.00  sec  25.0 MBytes   210 Mbits/sec    0   1.90 MBytes       
[  5]  25.00-26.00  sec  30.0 MBytes   252 Mbits/sec    0   1.90 MBytes       
[  5]  26.00-27.00  sec  36.2 MBytes   304 Mbits/sec    0   1.90 MBytes       
[  5]  27.00-28.00  sec  56.2 MBytes   472 Mbits/sec    0   1.90 MBytes       
[  5]  28.00-29.00  sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5]  29.00-30.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  30.00-31.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  31.00-32.00  sec  63.8 MBytes   534 Mbits/sec    0   1.90 MBytes       
[  5]  32.00-33.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  33.00-34.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  34.00-35.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  35.00-36.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  36.00-37.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  37.00-38.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  38.00-39.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  39.00-40.00  sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5]  40.00-41.00  sec  62.5 MBytes   525 Mbits/sec    0   1.90 MBytes       
[  5]  41.00-42.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  42.00-43.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  43.00-44.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  44.00-45.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  45.00-46.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  46.00-47.00  sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5]  47.00-48.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  48.00-49.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  49.00-50.00  sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5]  50.00-51.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  51.00-52.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  52.00-53.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  53.00-54.00  sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5]  54.00-55.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  55.00-56.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  56.00-57.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  57.00-58.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  58.00-59.00  sec  67.5 MBytes   566 Mbits/sec    0   1.90 MBytes       
[  5]  59.00-60.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  60.00-61.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  61.00-62.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  62.00-63.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  63.00-64.00  sec  66.2 MBytes   555 Mbits/sec    0   1.90 MBytes       
[  5]  64.00-65.00  sec  65.0 MBytes   546 Mbits/sec    0   1.90 MBytes       
[  5]  65.00-66.00  sec  66.2 MBytes   555 Mbits/sec    0   1.90 MBytes       
[  5]  66.00-67.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  67.00-68.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  68.00-69.00  sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5]  69.00-70.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  70.00-71.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  71.00-72.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  72.00-73.00  sec  65.0 MBytes   543 Mbits/sec    0   1.90 MBytes       
[  5]  73.00-74.00  sec  66.2 MBytes   558 Mbits/sec    0   1.90 MBytes       
[  5]  74.00-75.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  75.00-76.00  sec  67.5 MBytes   566 Mbits/sec    0   1.90 MBytes       
[  5]  76.00-77.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  77.00-78.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  78.00-79.00  sec  36.2 MBytes   304 Mbits/sec    0   1.90 MBytes       
[  5]  79.00-80.00  sec  38.8 MBytes   325 Mbits/sec    0   1.90 MBytes       
[  5]  80.00-81.00  sec  35.0 MBytes   294 Mbits/sec    1   1.90 MBytes       
[  5]  81.00-82.00  sec  28.8 MBytes   241 Mbits/sec    0   1.90 MBytes       
[  5]  82.00-83.00  sec  31.2 MBytes   262 Mbits/sec    0   1.90 MBytes       
[  5]  83.00-84.00  sec  28.8 MBytes   241 Mbits/sec    0   1.90 MBytes       
[  5]  84.00-85.00  sec  28.8 MBytes   241 Mbits/sec    0   1.90 MBytes       
[  5]  85.00-86.00  sec  37.5 MBytes   315 Mbits/sec    0   1.90 MBytes       
[  5]  86.00-87.00  sec  53.8 MBytes   451 Mbits/sec    0   1.90 MBytes       
[  5]  87.00-88.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  88.00-89.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  89.00-90.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  90.00-91.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  91.00-92.00  sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5]  92.00-93.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  93.00-94.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  94.00-95.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  95.00-96.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  96.00-97.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  97.00-98.00  sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5]  98.00-99.00  sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5]  99.00-100.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 100.00-101.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 101.00-102.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 102.00-103.00 sec  65.0 MBytes   544 Mbits/sec    0   1.90 MBytes       
[  5] 103.00-104.00 sec  66.2 MBytes   557 Mbits/sec    0   1.90 MBytes       
[  5] 104.00-105.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 105.00-106.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 106.00-107.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 107.00-108.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 108.00-109.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 109.00-110.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 110.00-111.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 111.00-112.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 112.00-113.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 113.00-114.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 114.00-115.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 115.00-116.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 116.00-117.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 117.00-118.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 118.00-119.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 119.00-120.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 120.00-121.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 121.00-122.00 sec  65.0 MBytes   546 Mbits/sec    0   1.90 MBytes       
[  5] 122.00-123.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 123.00-124.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 124.00-125.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 125.00-126.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 126.00-127.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 127.00-128.00 sec  66.2 MBytes   554 Mbits/sec    0   1.90 MBytes       
[  5] 128.00-129.00 sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5] 129.00-130.00 sec  66.2 MBytes   558 Mbits/sec    0   1.90 MBytes       
[  5] 130.00-131.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 131.00-132.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 132.00-133.00 sec  66.2 MBytes   555 Mbits/sec    0   1.90 MBytes       
[  5] 133.00-134.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 134.00-135.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 135.00-136.00 sec  66.2 MBytes   556 Mbits/sec    0   1.90 MBytes       
[  5] 136.00-137.00 sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5] 137.00-138.00 sec  65.0 MBytes   545 Mbits/sec    0   1.90 MBytes       
[  5] 138.00-139.00 sec  36.2 MBytes   304 Mbits/sec    0   1.90 MBytes       
[  5] 139.00-140.00 sec  32.5 MBytes   273 Mbits/sec    0   1.90 MBytes       
[  5] 140.00-141.00 sec  35.0 MBytes   294 Mbits/sec    0   1.90 MBytes       
[  5] 141.00-142.00 sec  30.0 MBytes   252 Mbits/sec    0   1.90 MBytes       
[  5] 142.00-143.00 sec  30.0 MBytes   252 Mbits/sec    0   1.90 MBytes       
[  5] 143.00-144.00 sec  27.5 MBytes   231 Mbits/sec    0   1.90 MBytes       
[  5] 144.00-145.00 sec  28.8 MBytes   241 Mbits/sec    0   1.90 MBytes       
[  5] 145.00-146.00 sec  38.8 MBytes   325 Mbits/sec    0   1.90 MBytes       
[  5] 146.00-147.00 sec  40.0 MBytes   336 Mbits/sec    0   1.90 MBytes       
[  5] 147.00-148.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 148.00-149.00 sec  60.0 MBytes   503 Mbits/sec    0   1.90 MBytes       
[  5] 149.00-150.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 150.00-151.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 151.00-152.00 sec  60.0 MBytes   503 Mbits/sec    0   1.90 MBytes       
[  5] 152.00-153.00 sec  60.0 MBytes   503 Mbits/sec    0   1.90 MBytes       
[  5] 153.00-154.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 154.00-155.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 155.00-156.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 156.00-157.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 157.00-158.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 158.00-159.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 159.00-160.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 160.00-161.00 sec  60.0 MBytes   503 Mbits/sec    0   1.90 MBytes       
[  5] 161.00-162.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 162.00-163.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 163.00-164.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 164.00-165.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 165.00-166.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 166.00-167.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 167.00-168.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 168.00-169.00 sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5] 169.00-170.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 170.00-171.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 171.00-172.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 172.00-173.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 173.00-174.00 sec  60.0 MBytes   503 Mbits/sec    0   1.90 MBytes       
[  5] 174.00-175.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 175.00-176.00 sec  60.0 MBytes   503 Mbits/sec    0   1.90 MBytes       
[  5] 176.00-177.00 sec  63.8 MBytes   534 Mbits/sec    0   1.90 MBytes       
[  5] 177.00-178.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 178.00-179.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 179.00-180.00 sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5] 180.00-181.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 181.00-182.00 sec  62.5 MBytes   524 Mbits/sec    0   1.90 MBytes       
[  5] 182.00-183.00 sec  61.2 MBytes   514 Mbits/sec    0   1.90 MBytes       
[  5] 183.00-184.00 sec  63.8 MBytes   535 Mbits/sec    0   1.90 MBytes       
[  5] 184.00-185.00 sec  18.8 MBytes   157 Mbits/sec   58   1.39 MBytes       
[  5] 185.00-186.00 sec  57.5 MBytes   482 Mbits/sec    0   1.53 MBytes       
[  5] 186.00-187.00 sec  57.5 MBytes   482 Mbits/sec    2   1.15 MBytes       
[  5] 187.00-188.00 sec  53.8 MBytes   451 Mbits/sec    0   1.22 MBytes       
[  5] 188.00-189.00 sec  52.5 MBytes   440 Mbits/sec    0   1.26 MBytes       
[  5] 189.00-190.00 sec  53.8 MBytes   451 Mbits/sec    0   1.29 MBytes       
[  5] 190.00-191.00 sec  55.0 MBytes   461 Mbits/sec    0   1.31 MBytes       
[  5] 191.00-192.00 sec  56.2 MBytes   472 Mbits/sec    0   1.32 MBytes       
[  5] 192.00-193.00 sec  53.8 MBytes   451 Mbits/sec    0   1.32 MBytes       
[  5] 193.00-194.00 sec  57.5 MBytes   482 Mbits/sec    0   1.32 MBytes       
[  5] 194.00-195.00 sec  55.0 MBytes   461 Mbits/sec    0   1.34 MBytes       
[  5] 195.00-196.00 sec  58.8 MBytes   493 Mbits/sec    0   1.37 MBytes       
[  5] 196.00-197.00 sec  60.0 MBytes   504 Mbits/sec    0   1.39 MBytes       
[  5] 197.00-198.00 sec  60.0 MBytes   503 Mbits/sec    0   1.42 MBytes       
[  5] 198.00-199.00 sec  38.8 MBytes   325 Mbits/sec    0   1.43 MBytes       
[  5] 199.00-200.00 sec  33.8 MBytes   283 Mbits/sec    0   1.45 MBytes       
[  5] 200.00-201.00 sec  33.8 MBytes   283 Mbits/sec    0   1.53 MBytes       
[  5] 201.00-202.00 sec  28.8 MBytes   241 Mbits/sec    0   1.61 MBytes       
[  5] 202.00-203.00 sec  28.8 MBytes   241 Mbits/sec    0   1.72 MBytes       
[  5] 203.00-204.00 sec  32.5 MBytes   273 Mbits/sec    0   1.85 MBytes       
[  5] 204.00-205.00 sec  23.8 MBytes   199 Mbits/sec    0   1.85 MBytes       
[  5] 205.00-206.00 sec  36.2 MBytes   304 Mbits/sec    0   1.85 MBytes       
[  5] 206.00-207.00 sec  52.5 MBytes   440 Mbits/sec    0   1.85 MBytes       
[  5] 207.00-208.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 208.00-209.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 209.00-210.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 210.00-211.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 211.00-212.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 212.00-213.00 sec  66.2 MBytes   555 Mbits/sec    0   1.85 MBytes       
[  5] 213.00-214.00 sec  65.0 MBytes   546 Mbits/sec    0   1.85 MBytes       
[  5] 214.00-215.00 sec  63.8 MBytes   535 Mbits/sec    0   1.85 MBytes       
[  5] 215.00-216.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 216.00-217.00 sec  63.8 MBytes   535 Mbits/sec    0   1.85 MBytes       
[  5] 217.00-218.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 218.00-219.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 219.00-220.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 220.00-221.00 sec  63.8 MBytes   535 Mbits/sec    0   1.85 MBytes       
[  5] 221.00-222.00 sec  67.5 MBytes   566 Mbits/sec    0   1.85 MBytes       
[  5] 222.00-223.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 223.00-224.00 sec  65.0 MBytes   544 Mbits/sec    0   1.85 MBytes       
[  5] 224.00-225.00 sec  66.2 MBytes   557 Mbits/sec    0   1.85 MBytes       
[  5] 225.00-226.00 sec  62.5 MBytes   525 Mbits/sec    0   1.85 MBytes       
[  5] 226.00-227.00 sec  66.2 MBytes   555 Mbits/sec    0   1.85 MBytes       
[  5] 227.00-228.00 sec  65.0 MBytes   546 Mbits/sec    0   1.85 MBytes       
[  5] 228.00-229.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 229.00-230.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 230.00-231.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 231.00-232.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 232.00-233.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 233.00-234.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 234.00-235.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 235.00-236.00 sec  66.2 MBytes   555 Mbits/sec    0   1.85 MBytes       
[  5] 236.00-237.00 sec  65.0 MBytes   546 Mbits/sec    0   1.85 MBytes       
[  5] 237.00-238.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 238.00-239.00 sec  67.5 MBytes   566 Mbits/sec    0   1.85 MBytes       
[  5] 239.00-240.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 240.00-241.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 241.00-242.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 242.00-243.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 243.00-244.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 244.00-245.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 245.00-246.00 sec  66.2 MBytes   555 Mbits/sec    0   1.85 MBytes       
[  5] 246.00-247.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 247.00-248.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 248.00-249.00 sec  63.8 MBytes   535 Mbits/sec    0   1.85 MBytes       
[  5] 249.00-250.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 250.00-251.00 sec  67.5 MBytes   566 Mbits/sec    0   1.85 MBytes       
[  5] 251.00-252.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 252.00-253.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 253.00-254.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 254.00-255.00 sec  66.2 MBytes   556 Mbits/sec    0   1.85 MBytes       
[  5] 255.00-256.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 256.00-257.00 sec  65.0 MBytes   545 Mbits/sec    0   1.85 MBytes       
[  5] 257.00-258.00 sec  37.5 MBytes   314 Mbits/sec    1   1.41 KBytes       
[  5] 258.00-259.00 sec  20.0 MBytes   168 Mbits/sec  123   1.41 MBytes       
[  5] 259.00-260.00 sec  33.8 MBytes   283 Mbits/sec    0   1.52 MBytes       
[  5] 260.00-261.00 sec  28.8 MBytes   241 Mbits/sec    0   1.65 MBytes       
[  5] 261.00-262.00 sec  28.8 MBytes   241 Mbits/sec    0   1.73 MBytes       
[  5] 262.00-263.00 sec  25.0 MBytes   210 Mbits/sec    0   1.78 MBytes       
[  5] 263.00-264.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 264.00-265.00 sec  26.2 MBytes   220 Mbits/sec    0   1.81 MBytes       
[  5] 265.00-266.00 sec  31.2 MBytes   262 Mbits/sec    0   1.81 MBytes       
[  5] 266.00-267.00 sec  45.0 MBytes   378 Mbits/sec    0   1.81 MBytes       
[  5] 267.00-268.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 268.00-269.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 269.00-270.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 270.00-271.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 271.00-272.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 272.00-273.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 273.00-274.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 274.00-275.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 275.00-276.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 276.00-277.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 277.00-278.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 278.00-279.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 279.00-280.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 280.00-281.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 281.00-282.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 282.00-283.00 sec  61.2 MBytes   514 Mbits/sec    0   1.81 MBytes       
[  5] 283.00-284.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 284.00-285.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 285.00-286.00 sec  61.2 MBytes   514 Mbits/sec    0   1.81 MBytes       
[  5] 286.00-287.00 sec  58.8 MBytes   493 Mbits/sec    0   1.81 MBytes       
[  5] 287.00-288.00 sec  61.2 MBytes   514 Mbits/sec    0   1.81 MBytes       
[  5] 288.00-289.00 sec  58.8 MBytes   493 Mbits/sec    0   1.81 MBytes       
[  5] 289.00-290.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 290.00-291.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 291.00-292.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 292.00-293.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 293.00-294.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 294.00-295.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 295.00-296.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 296.00-297.00 sec  61.2 MBytes   514 Mbits/sec    0   1.81 MBytes       
[  5] 297.00-298.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 298.00-299.00 sec  58.8 MBytes   493 Mbits/sec    0   1.81 MBytes       
[  5] 299.00-300.00 sec  60.0 MBytes   503 Mbits/sec    0   1.81 MBytes       
[  5] 300.00-301.00 sec  58.8 MBytes   493 Mbits/sec    0   1.81 MBytes       
[  5] 301.00-302.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 302.00-303.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 303.00-304.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 304.00-305.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 305.00-306.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 306.00-307.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 307.00-308.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 308.00-309.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 309.00-310.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 310.00-311.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 311.00-312.00 sec  55.0 MBytes   460 Mbits/sec    0   1.81 MBytes       
[  5] 312.00-313.00 sec  53.8 MBytes   452 Mbits/sec    0   1.81 MBytes       
[  5] 313.00-314.00 sec  56.2 MBytes   471 Mbits/sec    0   1.81 MBytes       
[  5] 314.00-315.00 sec  55.0 MBytes   462 Mbits/sec    0   1.81 MBytes       
[  5] 315.00-316.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 316.00-317.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 317.00-318.00 sec  58.8 MBytes   493 Mbits/sec    0   1.81 MBytes       
[  5] 318.00-319.00 sec  36.2 MBytes   304 Mbits/sec    1   1.81 MBytes       
[  5] 319.00-320.00 sec  35.0 MBytes   294 Mbits/sec    0   1.81 MBytes       
[  5] 320.00-321.00 sec  33.8 MBytes   283 Mbits/sec    0   1.81 MBytes       
[  5] 321.00-322.00 sec  31.2 MBytes   262 Mbits/sec    0   1.81 MBytes       
[  5] 322.00-323.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 323.00-324.00 sec  32.5 MBytes   272 Mbits/sec    0   1.81 MBytes       
[  5] 324.00-325.00 sec  25.0 MBytes   210 Mbits/sec    0   1.81 MBytes       
[  5] 325.00-326.00 sec  36.2 MBytes   304 Mbits/sec    0   1.81 MBytes       
[  5] 326.00-327.00 sec  47.5 MBytes   398 Mbits/sec    0   1.81 MBytes       
[  5] 327.00-328.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 328.00-329.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 329.00-330.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 330.00-331.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 331.00-332.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 332.00-333.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 333.00-334.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 334.00-335.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 335.00-336.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 336.00-337.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 337.00-338.00 sec  63.8 MBytes   535 Mbits/sec    0   1.81 MBytes       
[  5] 338.00-339.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 339.00-340.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 340.00-341.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 341.00-342.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 342.00-343.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 343.00-344.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 344.00-345.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 345.00-346.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 346.00-347.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 347.00-348.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 348.00-349.00 sec  67.5 MBytes   566 Mbits/sec    0   1.81 MBytes       
[  5] 349.00-350.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 350.00-351.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 351.00-352.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 352.00-353.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 353.00-354.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 354.00-355.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 355.00-356.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 356.00-357.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 357.00-358.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 358.00-359.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 359.00-360.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 360.00-361.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 361.00-362.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 362.00-363.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 363.00-364.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 364.00-365.00 sec  67.5 MBytes   566 Mbits/sec    0   1.81 MBytes       
[  5] 365.00-366.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 366.00-367.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 367.00-368.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 368.00-369.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 369.00-370.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 370.00-371.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 371.00-372.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 372.00-373.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 373.00-374.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 374.00-375.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 375.00-376.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 376.00-377.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 377.00-378.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 378.00-379.00 sec  38.8 MBytes   325 Mbits/sec    0   1.81 MBytes       
[  5] 379.00-380.00 sec  33.8 MBytes   283 Mbits/sec    0   1.81 MBytes       
[  5] 380.00-381.00 sec  35.0 MBytes   294 Mbits/sec    0   1.81 MBytes       
[  5] 381.00-382.00 sec  26.2 MBytes   220 Mbits/sec    0   1.81 MBytes       
[  5] 382.00-383.00 sec  30.0 MBytes   251 Mbits/sec    0   1.81 MBytes       
[  5] 383.00-384.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 384.00-385.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 385.00-386.00 sec  33.8 MBytes   283 Mbits/sec    0   1.81 MBytes       
[  5] 386.00-387.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 387.00-388.00 sec  67.5 MBytes   566 Mbits/sec    0   1.81 MBytes       
[  5] 388.00-389.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 389.00-390.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 390.00-391.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 391.00-392.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 392.00-393.00 sec  66.2 MBytes   555 Mbits/sec    0   1.81 MBytes       
[  5] 393.00-394.00 sec  65.0 MBytes   546 Mbits/sec    0   1.81 MBytes       
[  5] 394.00-395.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 395.00-396.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 396.00-397.00 sec  66.2 MBytes   555 Mbits/sec    0   1.81 MBytes       
[  5] 397.00-398.00 sec  65.0 MBytes   546 Mbits/sec    0   1.81 MBytes       
[  5] 398.00-399.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 399.00-400.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 400.00-401.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 401.00-402.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 402.00-403.00 sec  67.5 MBytes   566 Mbits/sec    0   1.81 MBytes       
[  5] 403.00-404.00 sec  66.2 MBytes   555 Mbits/sec    0   1.81 MBytes       
[  5] 404.00-405.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 405.00-406.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 406.00-407.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 407.00-408.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 408.00-409.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 409.00-410.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 410.00-411.00 sec  66.2 MBytes   555 Mbits/sec    0   1.81 MBytes       
[  5] 411.00-412.00 sec  65.0 MBytes   546 Mbits/sec    0   1.81 MBytes       
[  5] 412.00-413.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 413.00-414.00 sec  63.8 MBytes   535 Mbits/sec    0   1.81 MBytes       
[  5] 414.00-415.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 415.00-416.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 416.00-417.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 417.00-418.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 418.00-419.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 419.00-420.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 420.00-421.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 421.00-422.00 sec  63.8 MBytes   535 Mbits/sec    0   1.81 MBytes       
[  5] 422.00-423.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 423.00-424.00 sec  63.8 MBytes   535 Mbits/sec    0   1.81 MBytes       
[  5] 424.00-425.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 425.00-426.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 426.00-427.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 427.00-428.00 sec  66.2 MBytes   556 Mbits/sec    0   1.81 MBytes       
[  5] 428.00-429.00 sec  63.8 MBytes   535 Mbits/sec    0   1.81 MBytes       
[  5] 429.00-430.00 sec  66.2 MBytes   555 Mbits/sec    0   1.81 MBytes       
[  5] 430.00-431.00 sec  65.0 MBytes   546 Mbits/sec    0   1.81 MBytes       
[  5] 431.00-432.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 432.00-433.00 sec  63.8 MBytes   535 Mbits/sec    0   1.81 MBytes       
[  5] 433.00-434.00 sec  67.5 MBytes   566 Mbits/sec    0   1.81 MBytes       
[  5] 434.00-435.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 435.00-436.00 sec  65.0 MBytes   545 Mbits/sec    0   1.81 MBytes       
[  5] 436.00-437.00 sec  61.2 MBytes   514 Mbits/sec    0   1.81 MBytes       
[  5] 437.00-438.00 sec  61.2 MBytes   514 Mbits/sec    0   1.81 MBytes       
[  5] 438.00-439.00 sec  36.2 MBytes   304 Mbits/sec    0   1.81 MBytes       
[  5] 439.00-440.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 440.00-441.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 441.00-442.00 sec  25.0 MBytes   210 Mbits/sec    0   1.81 MBytes       
[  5] 442.00-443.00 sec  22.5 MBytes   189 Mbits/sec    0   1.81 MBytes       
[  5] 443.00-444.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 444.00-445.00 sec  25.0 MBytes   210 Mbits/sec    0   1.81 MBytes       
[  5] 445.00-446.00 sec  31.2 MBytes   262 Mbits/sec    0   1.81 MBytes       
[  5] 446.00-447.00 sec  45.0 MBytes   378 Mbits/sec    0   1.81 MBytes       
[  5] 447.00-448.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 448.00-449.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 449.00-450.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 450.00-451.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 451.00-452.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 452.00-453.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 453.00-454.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 454.00-455.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 455.00-456.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 456.00-457.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 457.00-458.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 458.00-459.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 459.00-460.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 460.00-461.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 461.00-462.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 462.00-463.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 463.00-464.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 464.00-465.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 465.00-466.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 466.00-467.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 467.00-468.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 468.00-469.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 469.00-470.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 470.00-471.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 471.00-472.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 472.00-473.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 473.00-474.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 474.00-475.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 475.00-476.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 476.00-477.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 477.00-478.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 478.00-479.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 479.00-480.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 480.00-481.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 481.00-482.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 482.00-483.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 483.00-484.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 484.00-485.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 485.00-486.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 486.00-487.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 487.00-488.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 488.00-489.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 489.00-490.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 490.00-491.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 491.00-492.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 492.00-493.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 493.00-494.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 494.00-495.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 495.00-496.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 496.00-497.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 497.00-498.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 498.00-499.00 sec  35.0 MBytes   294 Mbits/sec    0   1.81 MBytes       
[  5] 499.00-500.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 500.00-501.00 sec  31.2 MBytes   262 Mbits/sec    0   1.81 MBytes       
[  5] 501.00-502.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 502.00-503.00 sec  25.0 MBytes   210 Mbits/sec    0   1.81 MBytes       
[  5] 503.00-504.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 504.00-505.00 sec  26.2 MBytes   220 Mbits/sec    0   1.81 MBytes       
[  5] 505.00-506.00 sec  27.5 MBytes   231 Mbits/sec    0   1.81 MBytes       
[  5] 506.00-507.00 sec  40.0 MBytes   336 Mbits/sec    1   1.81 MBytes       
[  5] 507.00-508.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 508.00-509.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 509.00-510.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 510.00-511.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 511.00-512.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 512.00-513.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 513.00-514.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 514.00-515.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 515.00-516.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 516.00-517.00 sec  52.5 MBytes   441 Mbits/sec    0   1.81 MBytes       
[  5] 517.00-518.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 518.00-519.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 519.00-520.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 520.00-521.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 521.00-522.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 522.00-523.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 523.00-524.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 524.00-525.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 525.00-526.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 526.00-527.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 527.00-528.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 528.00-529.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 529.00-530.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 530.00-531.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 531.00-532.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 532.00-533.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 533.00-534.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 534.00-535.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 535.00-536.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 536.00-537.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 537.00-538.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 538.00-539.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 539.00-540.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 540.00-541.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 541.00-542.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 542.00-543.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 543.00-544.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 544.00-545.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 545.00-546.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 546.00-547.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 547.00-548.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 548.00-549.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 549.00-550.00 sec  53.8 MBytes   451 Mbits/sec    0   1.81 MBytes       
[  5] 550.00-551.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 551.00-552.00 sec  52.5 MBytes   440 Mbits/sec    0   1.81 MBytes       
[  5] 552.00-553.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 553.00-554.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 554.00-555.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 555.00-556.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 556.00-557.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 557.00-558.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 558.00-559.00 sec  35.0 MBytes   294 Mbits/sec    0   1.81 MBytes       
[  5] 559.00-560.00 sec  30.0 MBytes   252 Mbits/sec    0   1.81 MBytes       
[  5] 560.00-561.00 sec  22.5 MBytes   189 Mbits/sec    0   1.81 MBytes       
[  5] 561.00-562.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 562.00-563.00 sec  23.8 MBytes   199 Mbits/sec    0   1.81 MBytes       
[  5] 563.00-564.00 sec  25.0 MBytes   210 Mbits/sec    0   1.81 MBytes       
[  5] 564.00-565.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 565.00-566.00 sec  28.8 MBytes   241 Mbits/sec    0   1.81 MBytes       
[  5] 566.00-567.00 sec  35.0 MBytes   294 Mbits/sec    0   1.81 MBytes       
[  5] 567.00-568.00 sec  50.0 MBytes   419 Mbits/sec    0   1.81 MBytes       
[  5] 568.00-569.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 569.00-570.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 570.00-571.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 571.00-572.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 572.00-573.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 573.00-574.00 sec  55.0 MBytes   461 Mbits/sec    0   1.81 MBytes       
[  5] 574.00-575.00 sec  57.5 MBytes   482 Mbits/sec    0   1.81 MBytes       
[  5] 575.00-576.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 576.00-577.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 577.00-578.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 578.00-579.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 579.00-580.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 580.00-581.00 sec  56.2 MBytes   472 Mbits/sec    0   1.81 MBytes       
[  5] 581.00-582.00 sec  42.5 MBytes   357 Mbits/sec    3   1.31 MBytes       
[  5] 582.00-583.00 sec  55.0 MBytes   461 Mbits/sec    0   1.45 MBytes       
[  5] 583.00-584.00 sec  56.2 MBytes   472 Mbits/sec    0   1.57 MBytes       
[  5] 584.00-585.00 sec  55.0 MBytes   461 Mbits/sec    0   1.66 MBytes       
[  5] 585.00-586.00 sec  56.2 MBytes   472 Mbits/sec    0   1.71 MBytes       
[  5] 586.00-587.00 sec  56.2 MBytes   472 Mbits/sec    0   1.77 MBytes       
[  5] 587.00-588.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 588.00-589.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 589.00-590.00 sec  55.0 MBytes   461 Mbits/sec    0   1.80 MBytes       
[  5] 590.00-591.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 591.00-592.00 sec  57.5 MBytes   482 Mbits/sec    0   1.80 MBytes       
[  5] 592.00-593.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 593.00-594.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 594.00-595.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 595.00-596.00 sec  47.5 MBytes   398 Mbits/sec    0   1.80 MBytes       
[  5] 596.00-597.00 sec  56.2 MBytes   472 Mbits/sec    0   1.80 MBytes       
[  5] 597.00-598.00 sec  52.5 MBytes   440 Mbits/sec    0   1.80 MBytes       
[  5] 598.00-599.00 sec  51.2 MBytes   430 Mbits/sec    0   1.80 MBytes       
[  5] 599.00-600.00 sec  51.2 MBytes   430 Mbits/sec    0   1.80 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-600.00 sec  33.1 GBytes   474 Mbits/sec  190             sender
[  5]   0.00-600.02 sec  33.1 GBytes   474 Mbits/sec                  receiver

iperf Done.

I started with 300 seconds on my first try. I did not trigger the problem. Then, as you can see above, I tried 600 seconds and am still not able to trigger the problem today. Next, 1200 seconds.

I have noted a periodic burst of retries which could be the result of many things. I wish I had something to analyze and test that goes more in-depth than iperf3. Suggestions?

After trying 1200 seconds, if it is not successful, then I may look at trying a non-DFS channel. Do you have high congestion levels on the channel you are using?

gifter77 commented 1 year ago

I let my router auto select channels. I'm on 52 at the moment using 80MHz band. There is no congestion on this channel.

morrownr commented 1 year ago

While doing a 1200 second run, it happened again. Unplugging and replugging the adapter did not get me connected. I think I need to find a keyboard, mouse and video cable.

We are making progress. At least I think we are duplicating the problem.

gifter77 commented 1 year ago

We are making progress. At least I think we are duplicating the problem.

Or you found a new one :) It's surprising that I only need a few minutes to reproduce it. Just need to turn on this webrtc service and connect to it while running iperf or doing a speedtest on the board.

morrownr commented 1 year ago

Or you found a new one :)

This is a possibility also. I haven't looked at the log yet as I keep running into my real life that calls for real work. I did round up a mouse and keyboard. Now to find a cable. I recently moved and you know the rest of the story.

It's surprising that I only need a few minutes to reproduce it.

That may be a hint. I just could not get it to happen until I pushed overclocking to the limits and know I think I have seen it twice but there is still the Pi4B USB3 hub chipset and driver. Right now, my bet is on the Pi.

gifter77 commented 1 year ago

So I was looking at the journalctl logs a bit more and it seems this stack trace only happens when trying to restart the driver (by unloading and reloading it) when connection is lost. I have a service on the machine that pings my router at regular intervals and restarts the mt7921u module if connection is lost. See the whole sequence of events below:

May 12 15:23:43 prusamini sonar[717]: Connection lost,<redacted> not reachable!
May 12 15:23:43 prusamini sonar[717]: Restarting network in 10 seconds.
May 12 15:24:03 prusamini kernel: usbcore: deregistering interface driver mt7921u
May 12 15:24:03 prusamini kernel: wlan0: deauthenticating from <redacted> by local choice (Reason: 3=DEAUTH_LEAVING)
May 12 15:24:08 prusamini kernel: mt7921u 2-1:1.0: Message 00020003 (seq 15) timeout
May 12 15:24:11 prusamini kernel: mt7921u 2-1:1.0: timed out waiting for pending tx
May 12 15:24:11 prusamini kernel: ------------[ cut here ]------------
May 12 15:24:11 prusamini kernel: WARNING: CPU: 2 PID: 46 at kernel/kthread.c:650 kthread_park+0xb0/0xd0
May 12 15:24:11 prusamini kernel: Modules linked in: xt_DSCP xt_tcpudp nft_compat nf_tables nfnetlink ctr aes_arm64 aes_generic libaes>
May 12 15:24:11 prusamini kernel: CPU: 2 PID: 46 Comm: kworker/u8:2 Tainted: G         C         6.4.0-rc1-v8-custom+ #1
May 12 15:24:11 prusamini kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
May 12 15:24:11 prusamini kernel: Workqueue: mt76 mt7921_mac_reset_work [mt7921_common]
May 12 15:24:11 prusamini kernel: pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
May 12 15:24:11 prusamini kernel: pc : kthread_park+0xb0/0xd0
May 12 15:24:11 prusamini kernel: lr : mt76u_stop_tx+0x224/0x358 [mt76_usb]
May 12 15:24:11 prusamini kernel: sp : ffffffc0085b3c60
May 12 15:24:11 prusamini kernel: x29: ffffffc0085b3c60 x28: ffffff80498c9b58 x27: ffffff80498c08a0
May 12 15:24:11 prusamini kernel: x26: 0000000000000000 x25: ffffff80498c2068 x24: ffffff80498c4820
May 12 15:24:11 prusamini kernel: x23: ffffff80498c2068 x22: ffffffd26dfe6c48 x21: ffffff80498c2048
May 12 15:24:11 prusamini kernel: x20: ffffff804991d300 x19: 0000000000000100 x18: 0000000000000000
May 12 15:24:11 prusamini kernel: x17: 0000000000000000 x16: ffffffd26d0b3908 x15: 0000007fb07f2848
May 12 15:24:11 prusamini kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
May 12 15:24:11 prusamini kernel: x11: 0000000000000000 x10: 0000000000001a80 x9 : ffffffd2167bc914
May 12 15:24:11 prusamini kernel: x8 : ffffff804085d8e0 x7 : ffffffffffd428c1 x6 : ffffffd26e6246f0
May 12 15:24:11 prusamini kernel: x5 : ffffffc0085b4000 x4 : ffffffd26e4b9000 x3 : 0000000000002800
May 12 15:24:11 prusamini kernel: x2 : 0000000000000000 x1 : 0000000000000004 x0 : ffffff80499c0000
May 12 15:24:11 prusamini kernel: Call trace:
May 12 15:24:11 prusamini kernel:  kthread_park+0xb0/0xd0
May 12 15:24:11 prusamini kernel:  mt76u_stop_tx+0x224/0x358 [mt76_usb]
May 12 15:24:11 prusamini kernel:  mt7921u_mac_reset+0x88/0x288 [mt7921u]
May 12 15:24:11 prusamini kernel:  mt7921_mac_reset_work+0xb4/0x1b8 [mt7921_common]
May 12 15:24:11 prusamini kernel:  process_one_work+0x208/0x488
May 12 15:24:11 prusamini kernel:  worker_thread+0x4c/0x430
May 12 15:24:11 prusamini kernel:  kthread+0x120/0x138
May 12 15:24:11 prusamini kernel:  ret_from_fork+0x10/0x20
May 12 15:24:11 prusamini kernel: ---[ end trace 0000000000000000 ]---
May 12 15:24:11 prusamini kernel: mt7921u 2-1:1.0: HW/SW Version: 0x8a108a10, Build Time: 20230331110902a
May 12 15:24:11 prusamini kernel: mt7921u 2-1:1.0: WM Firmware Version: ____010000, Build Time: 20230331110939
May 12 15:24:13 prusamini kernel: wlan0: Driver requested disconnection from AP 00:00:00:00:00:00
May 12 15:24:15 prusamini kernel: mt7921u 2-1:1.0: timed out waiting for pending tx
May 12 15:24:15 prusamini dhcpcd[706]: wlan0: carrier lost

It seems that until my service tries to restart the driver, I just silently lose connection.

morrownr commented 1 year ago

Making more progress...

The mt7921u driver should not behave like it is even if it is not the initial cause of the series of events.

We need to find a way to get information about why the connection was lost. There are several others in this thread, maybe the can think of a way before we can.

Would you mind posting the contents of your service?

As I have time, I am still trying to find my HDMI cables for my Pi. Why the hell the foundation used those small ports is beyond me. Anyway, I will keep looking.

gifter77 commented 1 year ago

The service is a 3D printer thing: https://github.com/mainsail-crew/sonar

But I have modified the libs/core.sh to run this script when resetting the connection:

rmmod mt7921u
modprobe mt7921u
echo 3574 6211 > /sys/bus/usb/drivers/mt7921u/new_id
morrownr commented 1 year ago

I hope those commands are happening in serial, not parallel.

Nevertheless, the connection should not be dropped in the first place.

rjbrown99 commented 1 year ago

Also running into this, latest eeprom + firmware. May I ask one of you to post your hostapd config? Happy to test and share results but might be useful for us to at least be using the same AP config. Thanks!

morrownr commented 1 year ago

Hi @rjbrown99

May I ask one of you to post your hostapd config?

https://github.com/morrownr/USB-WiFi

Go down to menu item 9 and look under it for: Example - hostapd.conf for WiFi6 with mt7921au chipset based adapters

Several of us have been working on this example hostapd.conf and it seems to be on the way to being solid. It will have many of the settings for the mt7921au chipset included but then there are changes depending on your setup but you know that.

Also running into this...

Can you elaborate? Is it that the connectivity drops and stays dropped? On a Pi4B?

rjbrown99 commented 1 year ago

Thanks. In my case it's an RPi4 model B rev 1.4 with the NETGEAR Nighthawk WiFi 6E USB 3.0 Adapter (A8000) running the latest Raspberry Pi OS with updates. 6.1.21-v8+ kernel, current bootloader as of 1/11/23, stable release, latest eeprom, firmware ____010000-20230117170942 for the wifi chip. Oh and I'm on hostapd 2.10 from the bullseye-backports.

In my case hostapd just disappeared occasionally - stops broadcasting the SSID at all. Had not noticed a kernel panic or driver issue, hostapd log messages seem normal. Rebooting clears it.

Today I added "options mt76-usb disable_usb_sg=1" to a modprobe.d and confirmed it is active. Have not had it occur since then. I've also been hammering it with iperf3 consistent with your note above, no issues yet.

This is also why I asked for your config, I'll drop that in so I am closer to the rest of you in case it has something to do with my hostapd.

whitslack commented 1 year ago

firmware ____010000-20230117170942

@rjbrown99: That firmware is at least a couple of releases out of date. Check here for the latest firmware blobs if your distro is slow to distribute them.

morrownr commented 1 year ago

@rjbrown99

Today I added "options mt76-usb disable_usb_sg=1" to a modprobe.d and confirmed it is active.

Oh gee. I forgot to check on the scatter gather... I need to get it together. I've advocated taking scatter gather out for a long time based on the mt7612u chipset:

https://github.com/morrownr/7612u

@gifter77 Go down to Known Issue 2.

That firmware is at least a couple of releases out of date.

@whitslack has a sharp eye. The full instructions are available on the Main Menu, link in previous message. Look at item 8, section 2. mt7922 is section 1 and I actually have a PCIe card now based on the mt7922. It uses the mt7921e driver and it does support 160 MHz channel width.

morrownr commented 1 year ago

@gifter77

I just ran the following with no problems after disabling scatter gather:

$ iperf3 -c 192.168.1.1 -t 1800

rjbrown99 commented 1 year ago

Ah, interesting. I did copy down the following files to /lib/firmware/mediatek: BT_RAM_CODE_MT7922_1_1_hdr.bin WIFI_MT7922_patch_mcu_1_1_hdr.bin WIFI_RAM_CODE_MT7922_1.bin

Rebooted.. and still shows firmware-version: ____010000-20230117170942.

I then went and ran strings on all of the firmware files in that directory, and the only one showing that string was WIFI_RAM_CODE_MT7961_1.bin.

So I then replaced the 7961 driver files in /lib/firmware/mediatek: BT_RAM_CODE_MT7961_1_2_hdr.bin WIFI_MT7961_patch_mcu_1_2_hdr.bin WIFI_RAM_CODE_MT7961_1.bin

...and what do you know. Upon reboot: firmware-version: ____010000-20230331110939

So, at least in my case - the NETGEAR Nighthawk WiFi 6E USB 3.0 Adapter (A8000), which appears to use a mt7921u chipset, loads the 7961 driver files and not the 7922 files.

Thanks for the tip, let's see how it goes over the next few days with the latest firmware. Had I read the main menu more closely I would have figured that out sooner :)

gifter77 commented 1 year ago

@morrownr scatter gather has been disabled since the beginning for me. It doesn't fix the issue.

morrownr commented 1 year ago

@gifter77

@morrownr scatter gather has been disabled since the beginning for me. It doesn't fix the issue.

It wasn't disabled here. Once I disabled it, I cannot seem to get the connection to go down. I've been running iperf3 for 30 minutes at a time and the connection stays good. The problem I was seeing may have been related to scatter-gather. Still looking for my cable.

morrownr commented 1 year ago

@rjbrown99

So, at least in my case - the NETGEAR Nighthawk WiFi 6E USB 3.0 Adapter (A8000), which appears to use a mt7921u chipset, loads the 7961 driver files and not the 7922 files.

The Netgear A8000 uses the mt7921au chipset which is supported by the mt7921u driver. In the mt7921 family, there are also the mt7921, mt7921k and mt7922 which use the mt7921e driver.

It appears that the only difference between the mt7921 and mt7922 chipsets is that the mt7922 supports 160 MHz channel width whereas the mt7921 supports up to 80 MHz channel width. I am hoping to see a mt7922u chipset in usb wifi adapters later this year.

I received a new PCIe adapter based on the mt7922 chipset 3 days ago. I had not bought a PCI wifi card in over 20 years but I really wanted to check out driver support and see what it would do. It is plug and play on Ubuntu 23.04. It uses the mt7921e driver. It supports bluetooth as their is no USB3 to cause problems. It supports 160 MHz channel width. I did need to update the firmware:

BT_RAM_CODE_MT7922_1_1_hdr.bin WIFI_MT7922_patch_mcu_1_1_hdr.bin WIFI_RAM_CODE_MT7922_1.bin

As you found out, the firmware for the mt7921 is:

BT_RAM_CODE_MT7961_1_2_hdr.bin WIFI_MT7961_patch_mcu_1_2_hdr.bin WIFI_RAM_CODE_MT7961_1.bin

The mt7921u and mt7921e and mt7921s use the same mt7961 firmware

The mt7922 chipsets use the mt7922 firmware.

So, in my guide to installing firmware, the first section is for mt7922 chipsets. The second and third sections are about the mt7921, mt7921k, and mt7921au chipsets.

Is there anything else I can do to confuse you more? Well, Mediatek's numbering scheme is only something an engineer could like.

morrownr commented 1 year ago

I have continued pounding my Pi4B / CF-951AX and I can't get it to drop and fail to recover now that scatter-gather is turned off.

To me, it is obvious that scatter-gather support in mt76 is problematic. This is not new as many users have turned scatter-gather off for their mt7612u adapters when experiencing what I am seeing here. At one point, I did some testing with sg on and off to see what the difference in speed would be. If I recall correctly, I did not see any difference which caused me to ponder why it is on in the first place. I think it is time to do more testing to document the issue and then follow up with a patch or recommendation to the devs.

If anyone else has some time to also test with sg on and sg off, let me know and we can coordinate what we do and how to document it.

morrownr commented 1 year ago

scatter-gather test with mt7921au based adapter

Issue: connection drops and the only resolution is to reboot the system.

Raspberry Pi 4B RasPiOS 2023-05-03

I change the modulate parameter and rebooted between each test so as to alternate on and off.

iperf3 -c 192.168.1.1 -t 300

scatter-gather off (disable_usb_sg=1)

1:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec  19.9 GBytes   569 Mbits/sec    4             sender
[  5]   0.00-300.01 sec  19.9 GBytes   569 Mbits/sec                  receiver

2: 
[  5]   0.00-300.00 sec  19.9 GBytes   570 Mbits/sec    5             sender
[  5]   0.00-300.01 sec  19.9 GBytes   570 Mbits/sec                  receiver

3:
[  5]   0.00-300.00 sec  20.0 GBytes   573 Mbits/sec    2             sender
[  5]   0.00-300.01 sec  20.0 GBytes   573 Mbits/sec                  receiver

scatter-gather on (disable_usb_sg=0)

1:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec  19.9 GBytes   570 Mbits/sec    1             sender
[  5]   0.00-300.01 sec  19.9 GBytes   570 Mbits/sec                  receiver

2:
[  5]   0.00-300.00 sec  20.0 GBytes   572 Mbits/sec   48             sender
[  5]   0.00-300.01 sec  20.0 GBytes   572 Mbits/sec                  receiver

3.
[  5]   0.00-300.00 sec  19.9 GBytes   571 Mbits/sec    0             sender
[  5]   0.00-300.02 sec  19.9 GBytes   571 Mbits/sec                  receiver

Observation: So much for needing to average the results. I was careful to check that sg was on or off. I have no explanation for how the results could be so close. I see no evidence that sg is providing any performance increase.

Previous to this testing session, I have been able to see the issue of the connection being dropped and only a reboot will connect the situation. It happened twice a few days ago while testing with sg on. There is a history of this with mt7612u adapters. I have yet to duplicate the issue with sg off.

Conclusion: Further testing on different platforms is needed. I will test x86_64 next. Given the history of sg causing problems such as connections dropping that can only be corrected with a reboot, it may be better for the default to be disable_usb_sg=1 with a follow up to determine what the problem is.

rjbrown99 commented 1 year ago

The advice and input has been helpful, I'm up with a single A8000 in AP mode using the recompiled hostapd. Scatter/gather off, and has been giving me no problems or dropouts.

Instead of quitting while I'm ahead, I added a second A8000 to the same Pi. Why? I want one to act as an AP and the other to act as a client. This didn't work at all.

I can boot the system successfully and it sees both wlan1 and wlan2. I can bring up hostapd on wlan1 successfully. The moment I run an 'ifconfig wlan2 1.2.3.4', both adapters stop working. Dmesg output below.

The hassio references in the messages below are for a home assistant install running on the same Pi, which is all based around docker containers. I don't think it's immediately relevant to this but wanted to add a bit of context.

[   71.056709] mt7921u 2-2.1:1.0: vendor request req:63 off:d02c failed:-71
[   71.153775] mt7921u 2-2.1:1.0: vendor request req:63 off:d054 failed:-71
[   71.165759] usb 2-2: USB disconnect, device number 2
[   71.165771] usb 2-2.1: USB disconnect, device number 3
[   73.949825] mt7921u 2-2.4:1.0: Message 000008ed (seq 7) timeout
[   74.031835] ------------[ cut here ]------------
[   74.031860] WARNING: CPU: 0 PID: 39 at kernel/kthread.c:659 kthread_park+0xb0/0xc8
[   74.031896] Modules linked in: veth nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter bridge stp llc ipv6 cmac ctr aes_arm64 aes_generic libaes ccm overlay em28xx_rc si2157 lgdt3306a i2c_mux em28xx_dvb dvb_core hid_logitech_hidpp joydev nft_chain_nat xt_MASQUERADE xt_addrtype xt_nat nf_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp mt7921u xt_state xt_conntrack mt7921_common nf_conntrack mt76_connac_lib nf_defrag_ipv6 mt76_usb nf_defrag_ipv4 mt76 xt_HL nft_compat nf_tables em28xx nfnetlink mac80211 tveeprom hid_logitech_dj cdc_acm libarc4 brcmfmac brcmutil vc4 raspberrypi_hwmon v3d snd_soc_hdmi_codec drm_display_helper gpu_sched cec drm_dma_helper bcm2835_v4l2(C) drm_shmem_helper bcm2835_codec(C) bcm2835_isp(C) drm_kms_helper i2c_brcmstb snd_soc_core rpivid_hevc(C) bcm2835_mmal_vchiq(C) v4l2_mem2mem snd_compress videobuf2_vmalloc videobuf2_dma_contig videobuf2_memops snd_bcm2835(C) snd_pcm_dmaengine snd_pcm videobuf2_v4l2 videobuf2_common snd_timer snd videodev vc_sm_cma(C) syscopyarea mc
[   74.032224]  sysfillrect sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio cfg80211 drm i2c_dev i2c_bcm2835 fuse rfkill drm_panel_orientation_quirks backlight ip_tables x_tables
[   74.032292] CPU: 0 PID: 39 Comm: kworker/u8:1 Tainted: G         C         6.1.21-v8+ #1642
[   74.032304] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[   74.032312] Workqueue: mt76 mt7921_mac_reset_work [mt7921_common]
[   74.032357] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   74.032367] pc : kthread_park+0xb0/0xc8
[   74.032378] lr : mt76u_stop_rx+0x2c/0x98 [mt76_usb]
[   74.032409] sp : ffffffc00857bcf0
[   74.032414] x29: ffffffc00857bcf0 x28: 0000000000000000 x27: 0000000000000000
[   74.032431] x26: ffffff810a258998 x25: ffffff810a2508a0 x24: ffffffe8ca4ad858
[   74.032446] x23: ffffff810a258930 x22: 00000000ffffffed x21: ffffff810a252020
[   74.032462] x20: ffffff8107b18000 x19: ffffff810a252020 x18: 000000000000005e
[   74.032477] x17: 0000000000000000 x16: ffffffe8c96af4d8 x15: 0000000000000000
[   74.032492] x14: 0000000000000000 x13: 0000000000000001 x12: 0000000000000000
[   74.032506] x11: 0000000000000002 x10: 0000000000001a60 x9 : ffffffe87abce3ec
[   74.032522] x8 : ffffff81003cb980 x7 : ffffffc00857c000 x6 : ffffffc008578000
[   74.032536] x5 : ffffffc00857c000 x4 : 0000000000000000 x3 : 0000000000000001
[   74.032551] x2 : 0000000000000000 x1 : 0000000000000004 x0 : ffffff8108d08000
[   74.032565] Call trace:
[   74.032570]  kthread_park+0xb0/0xc8
[   74.032581]  mt76u_stop_rx+0x2c/0x98 [mt76_usb]
[   74.032604]  mt7921u_mac_reset+0x74/0x278 [mt7921u]
[   74.032625]  mt7921_mac_reset_work+0xa0/0x198 [mt7921_common]
[   74.032654]  process_one_work+0x208/0x480
[   74.032668]  worker_thread+0x50/0x428
[   74.032679]  kthread+0xfc/0x110
[   74.032690]  ret_from_fork+0x10/0x20
[   74.032702] ---[ end trace 0000000000000000 ]---
[   74.038347] wlan1: failed to remove key (1, ff:ff:ff:ff:ff:ff) from hardware (-19)
[   74.038397] wlan1: failed to remove key (4, ff:ff:ff:ff:ff:ff) from hardware (-19)
[   74.042381] mt7921u 2-2.4:1.0: chip reset failed
[   74.042479] wlan2: Driver requested disconnection from AP 00:00:00:00:00:00
[   74.138119] usb 2-2.4: USB disconnect, device number 4
[   74.374300] hassio: port 5(veth374241b) entered blocking state
[   74.374321] hassio: port 5(veth374241b) entered disabled state
[   74.374799] device veth374241b entered promiscuous mode
[   74.375610] hassio: port 5(veth374241b) entered blocking state
[   74.375643] hassio: port 5(veth374241b) entered forwarding state
[   74.543497] usb 2-2: new SuperSpeed USB device number 5 using xhci_hcd
[   74.647448] usb 2-2: New USB device found, idVendor=2109, idProduct=0813, bcdDevice=90.11
[   74.647479] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   74.647486] usb 2-2: Product: USB3.0 Hub
[   74.647491] usb 2-2: Manufacturer: VIA Labs, Inc.
[   74.650484] hub 2-2:1.0: USB hub found
[   74.650772] hub 2-2:1.0: 4 ports detected
[   75.042801] usb 2-2.1: new SuperSpeed USB device number 6 using xhci_hcd
[   75.069869] usb 2-2.1: New USB device found, idVendor=0846, idProduct=9060, bcdDevice= 1.00
[   75.069900] usb 2-2.1: New USB device strings: Mfr=2, Product=3, SerialNumber=4
[   75.069907] usb 2-2.1: Product: Wireless_Device
[   75.069912] usb 2-2.1: Manufacturer: MediaTek Inc.
[   75.069917] usb 2-2.1: SerialNumber: 000000000
[   75.070238] hassio: port 5(veth374241b) entered disabled state
[   75.163130] usb 2-2.1: reset SuperSpeed USB device number 6 using xhci_hcd
[   75.331666] mt7921u 2-2.1:1.0: HW/SW Version: 0x8a108a10, Build Time: 20230331110902a

[   75.344442] mt7921u 2-2.1:1.0: WM Firmware Version: ____010000, Build Time: 20230331110939
[   75.414485] usb 2-2.4: new SuperSpeed USB device number 7 using xhci_hcd
[   75.439697] usb 2-2.4: New USB device found, idVendor=0846, idProduct=9060, bcdDevice= 1.00
[   75.439727] usb 2-2.4: New USB device strings: Mfr=2, Product=3, SerialNumber=4
[   75.439735] usb 2-2.4: Product: Wireless_Device
[   75.439740] usb 2-2.4: Manufacturer: MediaTek Inc.
[   75.439745] usb 2-2.4: SerialNumber: 000000000
[   75.531247] usb 2-2.4: reset SuperSpeed USB device number 7 using xhci_hcd
[   75.699747] mt7921u 2-2.4:1.0: HW/SW Version: 0x8a108a10, Build Time: 20230331110902a
[   75.712988] mt7921u 2-2.4:1.0: WM Firmware Version: ____010000, Build Time: 20230331110939
[   75.816096] eth0: renamed from vethd1ef4ad
[   75.839222] hassio: port 5(veth374241b) entered blocking state
[   75.839250] hassio: port 5(veth374241b) entered forwarding state
[   80.861961] mt7921u 2-2.1:1.0: Message 00020003 (seq 15) timeout
[   80.862002] wlan1: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-110)
bjlockie commented 1 year ago

On 2023-05-16 13:43, Robert Brown wrote:

The moment I run an 'ifconfig wlan2 1.2.3.4', both adapters stop working. Dmesg output below.

Do they both have the same reg domain?

Do you have an adapter that uses a different driver you could try?

Do both work clients together?

Message ID: @.***>

morrownr commented 1 year ago

I'm up with a single A8000 in AP mode using the recompiled hostapd. Scatter/gather off, and has been giving me no problems or dropouts.

Cool. Making progress.

Instead of quitting while I'm ahead,

I have that problem also.

I added a second A8000 to the same Pi.

I smell power problems. Are you connecting the A8000's in the 2 USB3 ports on the Pi? What else is connected in USB ports on the Pi.

I want one to act as an AP and the other to act as a client. This didn't work at all.

That is a hard thing to do because networking is designed to prevent it. Using an ethernet cable or MoCA or Powerline AV2 is easy to setup but what you are doing is basically a repeater and that is hard.

The moment I run an 'ifconfig wlan2 1.2.3.4', both adapters stop working. Dmesg output below.

We need to prove that adequate power is available on USB. Remeber the USB subsystem on the Pi4B can only provide a max of 1200 mA and that is way below what the book says should be provided by 2 x USB2 and 2 x USB3. It should be around 2800 mA. The dmesg mess can be cause by inadequate power conditions.

whitslack commented 1 year ago

I want one to act as an AP and the other to act as a client. This didn't work at all.

That is a hard thing to do because networking is designed to prevent it. Using an ethernet cable or MoCA or Powerline AV2 is easy to setup but what you are doing is basically a repeater and that is hard.

Some clarification:
It's trivial to do this as long as you're not bridging the interfaces. (@rjbrown99 appears not to be attempting to bridge since they're assigning an IP address to the second interface.) In other words, if you're routing between the two interfaces, such that they're on different subnets, then everything will work fine, and there's no special magic required, assuming you've configured other clients on the second subnet with an appropriate route. Where things break (by design) is when you're trying to bridge, as then the Wi-Fi client would need to run in promiscuous mode to receive packets addressed to MAC addresses reachable via the opposite bridge port, but the AP to which that client is associated will not be sending those packets to that client, and the client can't pluck them out of the air directly from other clients, as the encryption keys will not match. To do it would require cooperation by the AP on that side, and that is indeed "hard."

rjbrown99 commented 1 year ago

My intention was to use one USB adatper as an AP, the other as a client. Neither of them bridged in any way, the Pi would be set up with packet forwarding enabled and basically acting as a router. The Pi would then run a series of intermediary services (like a Pihole for network-wide adblocking) for all of my clients.

The RPi may indeed be underpowered to make this happen, I have an unpowered USB3 hub that both adapters would have to connect to. I can't switch to a powered hub based upon limitations in wiring and power availability. I do have a few alternatives so I'll explore them first, but I think you are on to something.

morrownr commented 1 year ago

My intention was to use one USB adatper as an AP, the other as a client. Neither of them bridged in any way, the Pi would be set up with packet forwarding enabled and basically acting as a router.

Ah, I jumped the gun. My bad.

The RPi may indeed be underpowered to make this happen, I have an unpowered USB3 hub that both adapters would have to connect to.

If we knew what the power consumption is when pushing packets, then we would have a good idea. Being a USB3 device, the adapter is authorized up to 900 mA. Generally speaking, Mediatek adapters tend to use less power than Realtek adapters but I can't test a Netgear A8000 because I don't have one. Maybe I can dig out my tester and see what it says on another adapter with the same chipset. That would not be a perfect test but might give us an idea. I recently moved and many boxes are still unpacked so don't hold your breathe.

RasPi's are notorious for their USB subsystem not having much power and to top things off, powered hubs, for the most part, won't work well with the Pi's so simple projects that do not require much power are best. What you might experiment with is using the internal wifi in place of one of the adapters... on second thought, I just remembered all of the reasons I turn the internal wifi off.

I just remembered. With the following command, you can get how much power the adapter asks for:

lsusb -v

You have to scroll down until you find the adapter. It will look something like:

MaxPower 400mA

Generally this number is higher than what is actually used. The reading above is from my Alfa ACM (mt7612u, AC1200) adapter. I have measured power consumption and with iperf3 going, it actually uses about 360 mA. On the other hand the rtl8814au asks for 900 mA and it really does take that much... poor RasPi owners that try to use that adapter.

rjbrown99 commented 1 year ago

Thanks, interesting that the Netgear doesn't actually consume all that much power.

Bus 002 Device 003: ID 0846:9060 NetGear, Inc. Wireless_Device MaxPower 160mA

I do have a few other things plugged in that are consuming a fair amount of power, though. Media capture device is up to 500mA and Zigbee gateway is up to 100mA. Even the Logitech wireless keyboard/mouse receiver can consume up to 98mA.

From a different online article - "the maximum current the Raspberry Pi 4** USB ports (either combined or a single USB port) can pass through is 1.2A."

So just adding those up - 160mA Netgear 500mA video capture (Hauppauge USB) 100mA Zigbee 98mA Logitech wireless

That puts us at ~858mA. Adding another 160mA for a second Netgear wireless adapter would put us at 1.018A, which is getting pretty close to the limit of the Pi 4 hardware.

If I have time I'll retry with only the wifi adapters connected just to see if it exhibits the same error/panic or not. But I think we're probably just hitting the power limit of the Pi in my case.

morrownr commented 1 year ago

Bus 002 Device 003: ID 0846:9060 NetGear, Inc. Wireless_Device MaxPower 160mA

Wow. No wonder folks are reporting that it runs cool. I did do a thermal test on my little CF-951AX that uses the same chipset as your A8000. I ran iperf3 on it for 20 minutes and it was pumping around 630 Mbps and it was cool. I'm beginning to wonder how Mediatek accomplished this.

Adding another 160mA for a second Netgear wireless adapter would put us at 1.018A, which is getting pretty close to the limit of the Pi 4 hardware.

I forget the term but cables cause a little extra power to be needed so if there is a possibility to eliminate a cable or two, that might help but I like your idea of unplugging everything and test with just to two A8000's. There are ways to turn various other things that you don't need in Pi and that can save a little power overall.

Myself, I run my Pi's headless so I don't have to plug in a keyboard and mouse. By the way, I see you are using Logitech. Kernel 6.3 and later has a messed up Logitech driver so be aware of that, I found it the hard way.

For now, the last thing I will mention is that the USB3 hub chipset is, well, it has had a lot of bad words said about it since the Pi4B was released. I think @bjlockie said something recently that there might be another attempt at fixing things with a firmware update. I'm not holding my breathe. The reason I mention this is that when dealing with the USB subsystem in a Pi, especially the Pi4B, there are multiple causes of pain.

@morrownr

whitslack commented 1 year ago

Generally this number is higher than what is actually used.

Agreed, and from a reputable company like Netgear or ALFA, that's most likely true. However, I have seen some low-quality USB devices whose device descriptors outright lie about their current requirements, or another very popular gaff (even among some reputable manufacturers) is to begin drawing close to the advertised current as soon as the device is plugged in, despite that the device is supposed to limit its current draw to a very low level (I think something like 50 mA, but don't quote me on that) until it has successfully negotiated with the USB host to draw more. So, probably the Netgear adapter in this case is staying under its advertised current, but we might suspect some of those other devices on the bus. The only way to know for sure is to attach an inline USB current meter to audit each device's current draw while in use.

morrownr commented 1 year ago

The only way to know for sure is to attach an inline USB current meter to audit each device's current draw while in use.

Indeed. I have one of those inline USB current meters but it has not been unpacked yet. I'll try to get a better handle on this mt7921au chipset when able.

morrownr commented 1 year ago

@gifter77

The last msg from you said things were looking up after you turned scatter-gather off. Are you good to go now?

gifter77 commented 1 year ago

The last msg from you said things were looking up after you turned scatter-gather off. Are you good to go now?

Nop my last message said I had scatter-gather turned off from the beginning and this didn't fix the issue :)