search

gnuton / asuswrt-merlin.ng

Extends the support of Merlin firmware to more ASUS routers
Other
1.3k stars 75 forks source link

ZenWiFi XT8 v1 only 3 of 4 CPU cores in use [Updated] #500

Open robbbaxley opened 7 months ago

robbbaxley commented 7 months ago

Router Model Affected Models: ZenWiFi XT8 v1

Firmware Version Affected RT-AX95Q_3004_388.4_0-gnuton1_puresqubi

Is this bug present in upstream Merlin releases too? No

Describe the bug A few days ago, I reported that only 3 of 4 CPU cores are being utilized in this firmware version. All cores are being utilized in all releases of the official Asus firmware. I was doing some additional analysis and testing and found that when I transferred data to a USB drive (SSD) connected to the USB port on the router, 3 of the 4 cores would operate at or near 100% with the 4th core finally showing some utilization but only 3% to 5%. Transfer speed was slow at an average of 33Mb/s.

To Reproduce Steps to reproduce the behavior:

  1. Go to 'Network Map'
  2. Click on 'Status' tab
  3. Watch CPU utilization graph
  4. See error

Expected behavior A 4 CPU cores to be consistently utilized and better transfer speed to USB drive.

Screenshots Attached Web capture_25-11-2023_131541_10 0 1 200

dacootmeister commented 7 months ago

It's long known issue, no fix until now The 4th core is quieter then the first 3, but low level it still does something from linux htop: image

robbbaxley commented 7 months ago

It's odd, though, that all 4 cores are move evenly utilized on the standard Asus WRT firmware and I never saw utilization spike to (and stay at or near 100% utilization. To me, it appears that threading/scheduling is somehow different.

jazha commented 7 months ago

While transferring files internally on my network, from a wired source to a wifi destination, I get life from core 4 in gui.

AX8

robbbaxley commented 7 months ago

This is so odd!

Can I ask which version of firmware you are using? Do you know which version of hardware you have? Mine is v1 and I see utilization similar to what you are showing, but only when using the standard firmware. I have verified standard core utilization on Asus 3.0.0.4.388.23285 firmware.

jazha commented 7 months ago

Im on a V1 aswell.

I run 3004.388.4_0-gnuton1

Not sure how to extract the exact numbers, but its the only one gnuton released for 3004 series of FW I believe

robbbaxley commented 7 months ago

Web capture_30-11-2023_13492_10 0 1 200-1

I took this capture just now. The only time I see any utilization at all on Core-4 is when I am transferring data to the USB disk over the network. The first 3 cores work at nearly 100% during the transfer but Core-4 is barely utilized. I have 5 devices, all AP's and all connected over Wired backhaul (1 device is currently offline due to a backhaul failure). I have 3 XT8's and 2 XT9's. All version 1's. The XT9's are using the default Asus firmware.

jazha commented 7 months ago

Could be core is locked to specific tasks, like running ethernet/wifi connections etc. I dont know enough to figure that out. I wouldnt worry about low load unless you have issues with the router not working properly. Someone more familiar with the workings could probably shed some light on the 4th cores tasks?

robbbaxley commented 7 months ago

Here is another test I completed just now. First screen capture is after plugging in a USB drive. Nothing else. Notice how CPU utilization increased though I don't know why. In the second screen capture, it shows data transfer over the wired network to the USB drive. CPU utilization nearly maxes out the first 3 cores. The 4th core finally shows some utilization but only just.

Web capture_30-11-2023_14140_10 0 1 200

Web capture_30-11-2023_141227_10 0 1 200

robbbaxley commented 7 months ago

I should add, my only concern is that the data transfer rate over the wired network to the USB drive is much slower than expected.

jazha commented 7 months ago

What type of drive do you have connected? Maker/model/connection/portable/powered/size/file system etc?

Could be something as simple as different drives just work differently unfortunately.

robbbaxley commented 7 months ago

In the very first screen capture, for that test I used a 7.68TB Micron 5300 Pro SSD (550MB/s write speed). In the most recent test, I used a PNY 512GB Elite-X Fit USB 3.1 Flash Drive (200MB/s write speed).

Both drives had a new partition created and formatted NTFS in Windows 11 Pro. I did attempt to use the PNY drive as exFAT but the Asus router wouldn't use it with that file system.

jazha commented 7 months ago

For sure none of those should cause a slowdown in transfer speed. I have really no clue what throughoutput the router should in any given case. I feel you should get what the ethernet port can give, which should be just above 100MB/s transfer in "perfect" conditions. Personally I use a storage drive connected to a computer internally on the network, and not directly in the router, so I dont have any of the issues you mention.

robbbaxley commented 7 months ago

Yeah, I am stumped too. I tested my network card speed and it maxes out at just over 2.1Gb/s so I am confident that's not the bottleneck. I am pretty sure that the issue lies somewhere in the firmware. Since the 3 working cores max out I am reasonably confident that there is an issue with the USB and/or CPU threadding.

robbbaxley commented 7 months ago

Jazha, do you have a USB drive you can plug into the XT8 and see if you get the same results I am seeing?

jazha commented 7 months ago

Yeah, I am stumped too. I tested my network card speed and it maxes out at just over 2.1Gb/s so I am confident that's not the bottleneck. I am pretty sure that the issue lies somewhere in the firmware. Since the 3 working cores max out I am reasonably confident that there is an issue with the USB and/or CPU threadding.

Or worse, the HW is not good enough to support such speeds, resulting in offload to the cpu for things that should go via the storage controller (if I understand correctly good HW storage controllers should be capable of handling this easy, but maybe they put a cheap one in or did some other trick to keep costs down)

Jazha, do you have a USB drive you can plug into the XT8 and see if you get the same results I am seeing?

I would do this in a heartbeat if I wasnt running AMTM on it and I rely on the single port to hold an USB stick for the storage requirements for Diversion , Skynet etc.

soelu commented 7 months ago

image I am running the stock firmware still and see the same effect. So i assume this is not a Merlin but a Asus Stock Firmware "Issue"

Dodgydrains commented 7 months ago
image

Seems fine to me on stock 23285 FW. This is just running a speedtest on speedtest.net.

robbbaxley commented 7 months ago

@soelu, are you on hardware v1 or v2? I have v1 and, like @Dodgydrains, the cores operate properly on stock firmware for me.

Dodgydrains commented 7 months ago

@robbbaxley I am also on V1 hardware and stock FW and cores seem fine on it.

Dodgydrains commented 7 months ago
image

This seems to be working for me on latest 3004.388.5_0-gnuton0_beta1, I did dirty flash from stock 23285 FW.

soelu commented 7 months ago

image I have Hardware version V1. It uses the 4th CPU, but unproportionally. This is during a Zoom Call and running a speedtest in parallel.

gnuton commented 7 months ago

The load is not spread across all cpus, since cpu affinity kicks in to keep some processes in some threads. This has been manually fine tuned by ASUS and it's what you see in all ASUS routers.

This has been already discussed a while ago and RMerl has replied to this too with some tests. https://www.snbforums.com/threads/rt-ac66u_b1-not-using-2nd-cpu-core.39670/

soelu commented 7 months ago

So I guess this is a non-issue for the Merlin Firmware then?

gnuton commented 7 months ago

@soelu according to what I have read in the post there were not advantages. if you have specific applications which you wnna fine tune for using more threads you may try taskset

taskset -p 1 "$pid";

You can get the pid of your app running "ps" from the command line and you can replace 1 with the bitmask associated to the cores you want to use. You can also install irqbalance.

robbbaxley commented 7 months ago

I can also confirm that, as @Dodgydrains noted, the ZenWifi XT8 Beta firmware version RT-AX95Q_3004_388.5_0-gnuton0_beta1_puresqubi, is working much better as it pertains to thruput to the USB attached storage.

I would like to note the UL and DL appear to be backwards to me (I am uploading files to the USB drive from a PC). Any thoughts?

Web capture_13-12-2023_111135_10 0 1 200

Dodgydrains commented 7 months ago

I can also confirm that, as @Dodgydrains noted, the ZenWifi XT8 Beta firmware version RT-AX95Q_3004_388.5_0-gnuton0_beta1_puresqubi, is working much better as it pertains to thruput to the USB attached storage.

I would like to note the UL and DL appear to be backwards to me (I am uploading files to the USB drive from a PC). Any thoughts?

Web capture_13-12-2023_111135_10 0 1 200

Mine D/L and U/L seems right way round when doing a speedtest, but I did dirty flash over official FW 23285

robbbaxley commented 7 months ago

@Dodgydrains, I just checked and also verify that the Ul/DL looks correct when in internet download/streaming/speed-test. So the graph is probably correct and it just looks backwards to me.

Now if we could just figure out WHY on the Merlin firmware 3 cores are utilized heavily when, in the Asus released firmware, the load is much more evenly balanced across all 4 cores. At least USB transfers are working quite a bit better with this Beta firmware.

robbbaxley commented 6 months ago

I wanted to post an update regarding the new firmware version RT-AX95Q_3004_388.5_0-gnuton1_puresqubi.w.

Still showing 3 of 4 cores with high CPU utilization during file transfers to a USB drive connected to the primary node. The 4th core shows o% to 4% utilization during data transfer to USB storage. This is the same as with the previous firmware version.

I set the USB version to both USB 2.0 and USB 3.0 in the primary node (where the drive was connected) and tested with no change in transfer speed. All nodes connected via hardwire at 2.5Gb/s. Computers used to test was 1 Windows 11 desktop connected via 1Gb/s hardwire and 1 Linux laptop connected via wireless. There was no difference in CPU utilization on the primary node with the connected storage drive and no difference in data transfer rates.

LarryRosen108 commented 6 months ago

I have seen similar results with higher node 4 % but without any real impacts to usage of my 2 node XT8 environment. My transfer was done using winscp from a laptop with 1gb Ethernet connection directly to my main node that contains the USB 3.0 stick. This is on the latest .5 release.

Sent from my mobile device so please pardon autocorrect and brevity

On Thu, Jan 4, 2024 at 12:22 PM robbbaxley @.***> wrote:

I wanted to post an update regarding the new firmware version RT-AX95Q_3004_388.5_0-gnuton1_puresqubi.w.

Still showing 3 of 4 cores with high CPU utilization during file transfers to a USB drive connected to the primary node. The 4th core shows o% to 4% utilization during data transfer to USB storage. This is the same as with the previous firmware version.

I set the USB version to both USB 2.0 and USB 3.0 in the primary node (where the drive was connected) and tested with no change in transfer speed. All nodes connected via hardwire at 2.5Gb/s. Computers used to test was 1 Windows 11 desktop connected via 1Gb/s hardwire and 1 Linux laptop connected via wireless. There was no difference in CPU utilization on the primary node with the connected storage drive and no difference in data transfer rates.

— Reply to this email directly, view it on GitHub https://github.com/gnuton/asuswrt-merlin.ng/issues/500#issuecomment-1877484335, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2XR7N2TEWUW2M5STZPD2L3YM3QL7AVCNFSM6AAAAAA72QNZ56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGQ4DIMZTGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

swwwong commented 6 months ago

@robbbaxley

you can manaul adjust some task to the one with low usage .

cpu 3 2
doomwithdon commented 5 months ago

so this will always be an issue on gnuton builds only?

swwwong commented 5 months ago

@doomwithdon

I think this issue is from the asus-wrt fiwmware.

Befor install gnuton builds,CPU 3 stays idle.

it only use cpu 3 will used when I config an second open vpn client in gnuton builds.

vpn
davidpalmeruk commented 5 months ago

I came to add a vote to support XT9 and spotted this - my XT9 has this underutilised CPU4 so can only agree with comments that it seems to be just part of the Asus stock firmware.