[Bug]: Downloading audiobooks - poor performance on local LAN

codhopper commented 2 weeks ago

What happened?

I've been fighting with downloading audiobooks to my android phone for a while. Initially I thought it might have been my wifi, router, virtualisation layers, bug with pfsense, haproxy slowness etc.

Attempted to download them from a directly connected machine with 10gbit connection. iperf gets around 9.37gbit. Audiobook gets around 100mbit.

The CPU usage is also very high. Out of 2 or 4 cores allocated on the i5-10400 it seemed to use around 100% of a single core any way.

What did you expect to happen?

Audiobooks download in less than about 10 minutes. CPU load is lower.

Steps to reproduce the issue

Download large audiobook
Grab some coffee
Finish download

Audiobookshelf version

v2.10.1

How are you running audiobookshelf?

Docker

What OS is your Audiobookshelf server hosted from?

Linux

If the issue is being seen in the UI, what browsers are you seeing the problem on?

None

Logs

zlib set to 9
user@server-on-lan:~$ curl -o /dev/null http://192.168.70.2:10228/api/items/4f7088d8-3a90-4b95-b9df-8e2c13c7ba0a/download?token=eyJiwiatruncatedforj85I
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1265M    0 1265M    0     0  14.3M      0 --:--:--  0:01:28 --:--:-- 15.4M

zlib set to 1
user@server-on-lan:~$ curl -o /dev/null http://192.168.70.2:10228/api/items/4f7088d8-3a90-4b95-b9df-8e2c13c7ba0a/download?token=eyJiwiatruncatedforj85I
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1279M    0 1279M    0     0  30.5M      0 --:--:--  0:00:41 --:--:-- 30.5M

zlib set to 0
user@server-on-lan:~$ curl -o /dev/null http://192.168.70.2:10228/api/items/4f7088d8-3a90-4b95-b9df-8e2c13c7ba0a/download?token=eyJiwiatruncatedforj85I
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1368M    0 1368M    0     0   168M      0 --:--:--  0:00:08 --:--:--  173M

Additional Notes

audiobookshelf hosted on a machine with the CPU "AMD GX-415GA SOC with Radeon(tm) HD Graphics" performed quite a lot worse. Around 5-6MB/s (around 60mbit). Current machine is running a "Intel(R) Core(TM) i5-10400"

Adjusting the zlib compression level in /server/utils/zipHelpers.js speeds up the connection significantly (at the price of file size). It might even be decompressed on the app side? Most of my testing has been on the desktop.

codhopper commented 2 weeks ago

After a bit more investigation it looks like the app (v0.9.74-beta on android) does store the files unzipped. So the compression is used to concatenate files for transfer (and reduce bandwidth a bit).

The "bad performance" also seems to come a bit from the way nodejs is handling multi-threading. There might be a solution with both compression and reasonable performance, if the compression could fill a buffer or better share resources. From what I saw there were 4 worker threads (with 2 allocated virtual CPUs) causing quite a bit of context switching during the download.

advplyr commented 2 weeks ago

Are you using the Abs android mobile app to download or are you using the web client to download?

The web client download button zips all the files and downloads. The android/ios mobile app downloads the audio files and cover image individually and doesn't do any file compression.

codhopper commented 2 weeks ago

Are you using the Abs android mobile app to download or are you using the web client to download?

The web client download button zips all the files and downloads. The android/ios mobile app downloads the audio files and cover image individually and doesn't do any file compression.

Using a linux server with curl for testing (as it bypassed the wifi, haproxy and the router). I haven't tried downloading a larger audiobook via the app lately. I did download a 300Mb m4b one recently and it did take a while compared to what I would expect (hope for). Possibly 2 minutes or so.

nichwall commented 2 weeks ago

Is the app accessing the server directly over LAN (either using direct IP or NAT Loopback)? If the request is leaving your network before getting to the server you'll be limited by your Internet upload speed.

codhopper commented 2 weeks ago

I've done a few tests with the app on my phone:

WIFI speed around my house is much lower than I thought. Can get a solid connection at 140mbit, but it went as low as 32mbit in a bad position when connected to the distant access point. And in an average position it was getting 60-80mbit.
1.37GB audiobook downloaded with a stopwatch in 2 separate tests. 1min 48seconds and 2min 2seconds. (around 90-100mbit)
CPU utilisation when downloading from the app was certainly much lower. Around 6-9% of the i5-10400 core.
Downloading the zip file from the browser seems like a very uncommon use case, but can be significantly limited by CPU power (8s with zlib level 0 vs 1m28s with zlib level 9)

Summary: My wifi is slow.

Just to help clarify:

The curl output was from a machine on the local wired network, direct connection, downloading with the link from the web interface (compressed)
10gbit connection to the docker host, tested with iperf3 at around 9.37gbit
original phone tests were not very scientific
Migrated from the AMD GX-415GA to the i5-10400 and performance did improve a bit, but not very much. Almost all of my testing was done from a laptop with a gigabit NIC over the web interface (zip file download)

nichwall commented 5 days ago

From this discussion and additional research, I wonder if just removing the compression from the download zip is worth it, because compressing compressed files (such as mp3 or m4b, and then jpeg covers) on average doesn't save a lot of bandwidth and increases CPU load (as originally mentioned at the beginning of this discussion).

Could also be a Boolean server setting of "compress zip downloads" enabled by default with a note that the setting reduces file size but makes the download take longer?

advplyr commented 5 days ago

I used the compression level that was in the example from node-archiver, so no thought was put into that on my part. We can reduce it but it would be nice to get some general benchmarks

advplyr / audiobookshelf