advplyr / audiobookshelf

Self-hosted audiobook and podcast server
https://audiobookshelf.org
GNU General Public License v3.0
5.77k stars 393 forks source link

[Bug]: Loading items a lot slower than pre-2.4.0 versions #2073

Open ScuttleSE opened 10 months ago

ScuttleSE commented 10 months ago

Describe the issue

When loading for example my Library page, the cover art takes several seconds to loads. The nodejs-process on the server seems to spike on CPU-usage.

I did a profile in Chrome, and as is evident, the browser waits for several seconds to get the cover-images from the server. (pixelated for privacy)

image

Steps to reproduce the issue

  1. Load a library with a bunch of books that has coverart

Audiobookshelf version

2.4.1

How are you running audiobookshelf?

Docker

ScuttleSE commented 10 months ago

Okay, after some more clicking around, this is not just the covers...

After loading my library and then scrolling down about 60% of the list, it took ~140 seconds for the server to load up the book titles and covers.

DocDrydenn commented 10 months ago

Same here.

skyzuma commented 10 months ago

same here

Spinonduality commented 10 months ago

Same here - also it seems to stop matching after ~95 titles on 2.4.1 - didn't have any issues with matching on 2.4.0

stefanop1 commented 10 months ago

Same here - Also It doesn't follow the rule of not showing a series if it has only one book

barelylit commented 10 months ago

Massive slow down on this latest version for me too

nichwall commented 10 months ago

The reason the server is so much slower now is due to the entire database no longer being loaded into memory and is being queried to get information. The purpose is to improve performance on large libraries (especially if the database is larger than the available memory), but the queries need to be optimized. Advplyr has already improved the performance of some queries, but is taking some time to work on the apps so it'll be a bit before the next release.

It is safe to roll back to 2.3.3 without any additional steps to wait for 2.4.2 2.4.3 2.4.4 a future release if this makes things unusable. It is always a good idea to make a backup just in case.

Hallo951 commented 10 months ago

Is the problem solved with version 2.4.2?

nichwall commented 10 months ago

Is the problem solved with version 2.4.2?

No, 2.4.2 ended up being some quick bug fixes that were causing crashes. The performance issues still need to be addressed.

BlackHoleFox commented 9 months ago

For whatever its worth as a +1, I am also running into severe slowdowns related to loading cover images. A hard refresh of the library page takes ~40 seconds to finish loading on my local network due to slowdowns. As shown in these Firefox network request logs, each cover image is taking over 30 seconds to finish. Everything else is done long before:

slow

If I take one of these image URLs and load it on my own, it takes 800ms-1s to load, which is still pretty bad considering that these are just being streamed off disk (in all of my testing, the cover metadata cache is already fully populated). Given that it only gets slower when multiple are requested at once (1s -> 30s), maybe there's a contention bug somewhere?

The reason the server is so much slower now is due to the entire database no longer being loaded into memory and is being queried to get information.

This is definitely not the case for cover images at least. The files aren't stored in the database and the load times are too slow to just be a "read DB from disk" issue.

nichwall commented 9 months ago

If I take one of these image URLs and load it on my own, it takes 800ms-1s to load, which is still pretty bad considering that these are just being streamed off disk (in all of my testing, the cover metadata cache is already fully populated). Given that it only gets slower when multiple are requested at once (1s -> 30s), maybe there's a contention bug somewhere?

What URL are you using to load it directly? Mine includes a call to the API (and the corresponding library item with a token), so to me it makes sense that the queries being slow within the API are also slow to load the cover images. Granted I deal more with hardware than internet stuff, so I could be misunderstanding the route the data is taking.

haptiqblack commented 9 months ago

Just notifying I am still seeing the cover art load slowly after update 2.4.3. Doesn't appear to have changed things for my instance.

nichwall commented 9 months ago

Just notifying I am still seeing the cover art load slowly after update 2.4.3. Doesn't appear to have changed things for my instance.

The last few patches have mostly been crash fixes, so the performance has not been fully addressed yet. The performance fixes will be major enough that they will be called out in the release notes, and I (or somebody else) will comment here as well.

Thanks for your patience. You can always revert to 2.3.3 if you want to wait for the performance fixes in a future release.

haptiqblack commented 9 months ago

Just notifying I am still seeing the cover art load slowly after update 2.4.3. Doesn't appear to have changed things for my instance.

The last few patches have mostly been crash fixes, so the performance has not been fully addressed yet. The performance fixes will be major enough that they will be called out in the release notes, and I (or somebody else) will comment here as well.

Thanks for your patience. You can always revert to 2.3.3 if you want to wait for the performance fixes in a future release.

No worries appreciate the info!

selfhost-alt commented 9 months ago

I did some simple investigation into this, and I found that all of the library item routes (including the one for fetching the cover) use this middleware, which calls LibraryItem.getOldById which runs this slow query: https://github.com/advplyr/audiobookshelf/blob/604f52762b14e5f64eedb56c136a8211be36cb2c/server/models/LibraryItem.js#L422-L454

That query ends up doing some nested queries to look up authors and book series, which at least for me seems to be a major contributor to the slowness of the query.

SELECT `libraryItem`.`id`, `libraryItem`.`ino`, `libraryItem`.`path`, `libraryItem`.`relPath`, ...
FROM `libraryItems` AS `libraryItem`

LEFT OUTER JOIN `books` AS `book` ON `libraryItem`.`mediaId` = `book`.`id`

LEFT OUTER JOIN (
  `bookAuthors` AS `book->authors->bookAuthor`
  INNER JOIN `authors` AS `book->authors`
  ON `book->authors`.`id` = `book->authors->bookAuthor`.`authorId`
) ON `book`.`id` = `book->authors->bookAuthor`.`bookId`

LEFT OUTER JOIN (
  `bookSeries` AS `book->series->bookSeries`
  INNER JOIN `series` AS `book->series`
  ON `book->series`.`id` = `book->series->bookSeries`.`seriesId`
) ON `book`.`id` = `book->series->bookSeries`.`bookId`

LEFT OUTER JOIN `podcasts` AS `podcast` ON `libraryItem`.`mediaId` = `podcast`.`id`

LEFT OUTER JOIN `podcastEpisodes` AS `podcast->podcastEpisodes` ON `podcast`.`id` = `podcast->podcastEpisodes`.`podcastId`

WHERE `libraryItem`.`id` = '8b9c8cb6-7900-46a2-9c75-7f74f0d298ba'
ORDER BY `book->authors->bookAuthor`.`createdAt` ASC, `book->series->bookSeries`.`createdAt` ASC;

When I run explain query plan I get the following:

id parent notused detail
3 0 0 MATERIALIZE 1
8 3 0 SCAN TABLE bookAuthors AS book->authors->bookAuthor
10 3 0 SEARCH TABLE authors AS book->authors USING INDEX sqlite_autoindex_authors_1 (id=?)
38 0 0 MATERIALIZE 2
43 38 0 SCAN TABLE bookSeries AS book->series->bookSeries
45 38 0 SEARCH TABLE series AS book->series USING INDEX sqlite_autoindex_series_1 (id=?)
78 0 0 SEARCH TABLE libraryItems AS libraryItem USING INDEX sqlite_autoindex_libraryItems_1 (id=?)
84 0 0 SEARCH TABLE books AS book USING INDEX sqlite_autoindex_books_1 (id=?)
93 0 0 SCAN SUBQUERY 1
118 0 0 SEARCH SUBQUERY 2 USING AUTOMATIC COVERING INDEX (bookId=?)
126 0 0 SEARCH TABLE podcasts AS podcast USING INDEX sqlite_autoindex_podcasts_1 (id=?)
161 0 0 SEARCH TABLE podcastEpisodes AS podcast->podcastEpisodes USING AUTOMATIC COVERING INDEX (podcastId=?)
299 0 0 USE TEMP B-TREE FOR ORDER BY

I'm not a SQL optimization expert, but I think the unfiltered scans of bookAuthors<->authors and bookSeries<->series are contributing to the query slowness.

advplyr commented 9 months ago

I replaced that API endpoint to get the cover so that it doesn't go through the regular middleware that as @selfhost-alt pointed out is loading unnecessary amounts of data.

When this is built if anyone wants to test on edge docker version we can determine if the covers were the main problem with the performance. I can see it being part of the problem but I think the main issue is the performance of the /libraries/:id/items and the /libraries/:id/personalized endpoints.

ScuttleSE commented 9 months ago

I did a quick test, and loading covers is faster, but there are still issues. I opened up my library, and the first 10 or so books visible on the page popped up fine, I then scrolled about 2/3 down the list (~1400 books), and it took around 20-30 seconds for the titles of the books to load and then another few seconds of the cover art to load.

This URL for example took 25 seconds to load: https://site.example.org/api/libraries/dac312d8-6b0e-4b3c-ad6e-8e50a9d8d8b1/items?sort=media.metadata.title&desc=0&limit=100&page=1&minified=1&include=rssfeed,numEpisodesIncomplete

There were 10 of these requests being done almost at the same time, with just the page-value increasing by 1, all of them taking 25-28 seconds to complete. After that there were two batches of coverart loaded. The first one took about 20 seconds, second one took about 5 seconds.

image

advplyr commented 9 months ago

@ScuttleSE Do you have the server setting ignore prefixes when sorting enabled? Currently that column in the db is not indexed. I was testing with a library of ~2700 books those queries take anywhere from 1 to 5 seconds to complete. When I disabled that server setting the queries were less than a second.

image

For testing I scrolled down the library page sorted by title at a fast speed and watched the server logs with debug logs turned on.

With ignore prefixes when sorting enabled: I'm only sharing the first half of the requests

[2023-09-22 10:34:26] DEBUG: Loaded 100 of 2680 items for libary page in 2.46s (LibraryItem.js:469)
[2023-09-22 10:34:31] DEBUG: Loaded 100 of 2680 items for libary page in 5.52s (LibraryItem.js:469)
[2023-09-22 10:34:31] DEBUG: Loaded 100 of 2680 items for libary page in 5.57s (LibraryItem.js:469)
[2023-09-22 10:34:32] DEBUG: Loaded 100 of 2680 items for libary page in 3.73s (LibraryItem.js:469)
[2023-09-22 10:34:33] DEBUG: Loaded 100 of 2680 items for libary page in 1.91s (LibraryItem.js:469)
[2023-09-22 10:34:36] DEBUG: Loaded 100 of 2680 items for libary page in 2.56s (LibraryItem.js:469)
[2023-09-22 10:34:36] DEBUG: Loaded 100 of 2680 items for libary page in 2.67s (LibraryItem.js:469)
[2023-09-22 10:34:40] DEBUG: Loaded 100 of 2680 items for libary page in 0.99s (LibraryItem.js:469)
[2023-09-22 10:34:46] DEBUG: Loaded 100 of 2680 items for libary page in 0.78s (LibraryItem.js:469)
[2023-09-22 10:34:52] DEBUG: Loaded 100 of 2680 items for libary page in 1.81s (LibraryItem.js:469)
[2023-09-22 10:34:55] DEBUG: Loaded 100 of 2680 items for libary page in 3.01s (LibraryItem.js:469)
[2023-09-22 10:34:56] DEBUG: Loaded 100 of 2680 items for libary page in 3.01s (LibraryItem.js:469)
[2023-09-22 10:34:58] DEBUG: Loaded 100 of 2680 items for libary page in 3.34s (LibraryItem.js:469)
[2023-09-22 10:34:58] DEBUG: Loaded 100 of 2680 items for libary page in 3.35s (LibraryItem.js:469)
[2023-09-22 10:35:02] DEBUG: Loaded 100 of 2680 items for libary page in 3.23s (LibraryItem.js:469)
[2023-09-22 10:35:02] DEBUG: Loaded 100 of 2680 items for libary page in 3.35s (LibraryItem.js:469)
[2023-09-22 10:35:07] DEBUG: Loaded 100 of 2680 items for libary page in 5.15s (LibraryItem.js:469)
[2023-09-22 10:35:09] DEBUG: Loaded 100 of 2680 items for libary page in 4.64s (LibraryItem.js:469)
[2023-09-22 10:35:09] DEBUG: Loaded 100 of 2680 items for libary page in 6.77s (LibraryItem.js:469)
[2023-09-22 10:35:10] DEBUG: Loaded 100 of 2680 items for libary page in 2.43s (LibraryItem.js:469)

With ignore prefixes when sorting disabled:

[2023-09-22 10:32:05] DEBUG: Loaded 100 of 2680 items for libary page in 0.72s (LibraryItem.js:469)
[2023-09-22 10:32:13] DEBUG: Loaded 100 of 2680 items for libary page in 0.56s (LibraryItem.js:469)
[2023-09-22 10:32:14] DEBUG: Loaded 100 of 2680 items for libary page in 0.72s (LibraryItem.js:469)
[2023-09-22 10:32:15] DEBUG: Loaded 100 of 2680 items for libary page in 0.67s (LibraryItem.js:469)
[2023-09-22 10:32:15] DEBUG: Loaded 100 of 2680 items for libary page in 0.35s (LibraryItem.js:469)
[2023-09-22 10:32:16] DEBUG: Loaded 100 of 2680 items for libary page in 0.72s (LibraryItem.js:469)
[2023-09-22 10:32:16] DEBUG: Loaded 100 of 2680 items for libary page in 0.71s (LibraryItem.js:469)
[2023-09-22 10:32:17] DEBUG: Loaded 100 of 2680 items for libary page in 0.86s (LibraryItem.js:469)
[2023-09-22 10:32:17] DEBUG: Loaded 100 of 2680 items for libary page in 0.40s (LibraryItem.js:469)
[2023-09-22 10:32:18] DEBUG: Loaded 100 of 2680 items for libary page in 0.56s (LibraryItem.js:469)
[2023-09-22 10:32:18] DEBUG: Loaded 100 of 2680 items for libary page in 0.82s (LibraryItem.js:469)
[2023-09-22 10:32:19] DEBUG: Loaded 100 of 2680 items for libary page in 0.64s (LibraryItem.js:469)
[2023-09-22 10:32:20] DEBUG: Loaded 100 of 2680 items for libary page in 0.76s (LibraryItem.js:469)
[2023-09-22 10:32:20] DEBUG: Loaded 100 of 2680 items for libary page in 0.92s (LibraryItem.js:469)
[2023-09-22 10:32:20] DEBUG: Loaded 100 of 2680 items for libary page in 0.87s (LibraryItem.js:469)
[2023-09-22 10:32:21] DEBUG: Loaded 100 of 2680 items for libary page in 0.60s (LibraryItem.js:469)
[2023-09-22 10:32:21] DEBUG: Loaded 100 of 2680 items for libary page in 0.72s (LibraryItem.js:469)
ScuttleSE commented 9 months ago

That does seem to be one of the contributing factors. After I disabled the prefixes-setting, the queries that earlier took 25-30 seconds now took 2-4 seconds.

ScuttleSE commented 9 months ago

I do wonder why my system is that much slower though...

[2023-09-22 16:07:57] DEBUG: Loaded 100 of 1450 items for libary page in 0.48s (LibraryItem.js:469)
[2023-09-22 16:08:09] DEBUG: Loaded 100 of 1450 items for libary page in 1.48s (LibraryItem.js:469)
[2023-09-22 16:08:09] DEBUG: Loaded 100 of 1450 items for libary page in 1.61s (LibraryItem.js:469)
[2023-09-22 16:08:09] DEBUG: Loaded 100 of 1450 items for libary page in 1.81s (LibraryItem.js:469)
[2023-09-22 16:08:09] DEBUG: Loaded 100 of 1450 items for libary page in 2.17s (LibraryItem.js:469)
[2023-09-22 16:08:09] DEBUG: Loaded 100 of 1450 items for libary page in 1.61s (LibraryItem.js:469)
[2023-09-22 16:08:10] DEBUG: Loaded 100 of 1450 items for libary page in 2.04s (LibraryItem.js:469)
[2023-09-22 16:08:10] DEBUG: Loaded 100 of 1450 items for libary page in 2.35s (LibraryItem.js:469)
[2023-09-22 16:08:10] DEBUG: Loaded 100 of 1450 items for libary page in 2.58s (LibraryItem.js:469)
[2023-09-22 16:08:10] DEBUG: Loaded 100 of 1450 items for libary page in 2.74s (LibraryItem.js:469)

That is with prefixes disabled Btw, no idea why my logs says "libary" and yours say "library"...

advplyr commented 9 months ago

Thanks for checking. Even the faster queries I showed are slower then they should be. Here are some things that need to be done to speed things up:

1) The queries need to be optimized to use indexes properly and reduce the number of subqueries 2) More data is being loaded then necessary. One example is the audioFiles column in the books table 3) After the data is loaded from the db it is converted to the old data model (this may take some more time to phase out)

advplyr commented 9 months ago

From my testing big improvements should be in v2.4.4

Collapse series on the library page will still be slow to the point where you might want to avoid that on large libraries. Let me know if you see improvement

BlackHoleFox commented 9 months ago

Updated with my library and saw a night to day improvement. The library screen loads in just a few seconds now instead of 30-70s. Great work so far :)

ScuttleSE commented 9 months ago

Hrm...I must say, I haven't noticed that much of an improvement.

If I open my library and then scroll down about 70%, it takes 30-40 seconds for both the book titles and covers to appear

This link for example took about 39 seconds to load:

https://abs.example.org/api/libraries/dac312d8-6b0e-4b3c-ad6e-8e50a9d8d8b1/items?sort=media.metadata.title&desc=0&limit=100&page=8&minified=1&include=rssfeed,numEpisodesIncomplete

ScuttleSE commented 9 months ago

If I disable Prefix-handling, it cuts down from around 30 seconds down to 10 doing the same.

Covers are still pretty slow, 10-12 seconds each

Dragonatorul commented 9 months ago

I'm seeing a similar issue in podcasts. Disabling or enabling the prefix option does not seem to make any noticeable difference.

The following logs are from switching between home, latest and library tabs.

audiobookshelf    | [2023-10-02 18:16:59] DEBUG: Loaded 4 personalized shelves in 16.56s (LibraryItem.js:667)
audiobookshelf    | [2023-10-02 18:18:42] DEBUG: Loaded 4 personalized shelves in 16.33s (LibraryItem.js:667)
audiobookshelf    | [2023-10-02 18:19:22] DEBUG: Loaded 100 of 131 items for libary page in 13.01s (LibraryItem.js:469)
audiobookshelf    | [2023-10-02 18:20:42] DEBUG: Loaded 4 personalized shelves in 16.93s (LibraryItem.js:667)
audiobookshelf    | [2023-10-02 18:21:06] DEBUG: Loaded 100 of 131 items for libary page in 12.99s (LibraryItem.js:469)
kaldigo commented 8 months ago

I am still having these issues including the slow loading covers. I did test a new install with the same library and it was much faster. Is there any way to migrate the data across without copying the database? Or maybe something that can be done to clean up the old database

djolker commented 7 months ago

I'm experiencing this issue in my Podcasts library but primarily only in relation to individual shows that have a very large number of episodes. When attempting to view all episodes of a show in my library with 1-100 episodes, the load time is near instant to maybe half a second. A show with 800+ episodes will take over 10 seconds to load.

Eftimin70 commented 7 months ago

I had the hope that a clean installation of version 2.5.0 and building up a fresh database could speed things up again. Unfortunitly that did't help and it still is very slow compared to 2.3.3. Especially the library view with 'Collapse Series' and series view are barely usable. So I have to stick with 2.3.3. and its quirks.

lkiesow commented 7 months ago

I've started looking into this as well. There seem to be several code paths which are still quite slow. Not unusable, but… annoying.

Loading covers is one of the things which are still quite slow. The following query seems to be the problem (taken when loading “Home”):

-- Ran the following query in 1117ms:
SELECT `libraryItem`.`id`,
       `libraryItem`.`mediatype`,
       `libraryItem`.`mediaid`,
       `libraryItem`.`libraryid`,
       `book`.`id`           AS `book.id`,
       `book`.`coverpath`    AS `book.coverPath`,
       `book`.`tags`         AS `book.tags`,
       `book`.`explicit`     AS `book.explicit`,
       `podcast`.`id`        AS `podcast.id`,
       `podcast`.`coverpath` AS `podcast.coverPath`,
       `podcast`.`tags`      AS `podcast.tags`,
       `podcast`.`explicit`  AS `podcast.explicit`
FROM   `libraryitems` AS `libraryItem`
       LEFT OUTER JOIN `books` AS `book`
                    ON `libraryItem`.`mediaid` = `book`.`id`
       LEFT OUTER JOIN `podcasts` AS `podcast`
                    ON `libraryItem`.`mediaid` = `podcast`.`id`
WHERE  `libraryItem`.`id` = '9217841f-400a-490b-a10f-e4529066c07d';

One problem here could be the mismatching data types of the joined columns (see #2324). I'm not sure about SQLite UUID and UUIDv4, but in general, a data type mismatch can make a join really expensive. After all, the database might need to perform a type conversion on the fly.

That being said, I'm wondering if we need the joins at all. Looking at the data being used, we have libraryId from the table libraryItems and explicit, tags and coverPath from either books or podcasts. This means that all the latter three fields exist in both media type tables. If we move the common columns to the libraryItems table instead, we would need no join at all. That would probably speed up the whole process quite a bit.

If that's not possible for any reason, we could also try splitting up the query and running then as separate queries. This might still cause less data to be searched ovverall and be faster.

advplyr commented 7 months ago

That being said, I'm wondering if we need the joins at all. Looking at the data being used, we have libraryId from the table libraryItems and explicit, tags and coverPath from either books or podcasts. This means that all the latter three fields exist in both media type tables. If we move the common columns to the libraryItems table instead, we would need no join at all. That would probably speed up the whole process quite a bit.

Yeah I plan on doing this but right now the clients are sending a library item id, not a media id. I wanted to update the clients and the API route to use media id.

lkiesow commented 7 months ago

So you want the clients to request something like /api/books/{mediaId}/cover and /api/podcasts/{mediaId}/cover and then just access one of the tables? That would also definitely work and be faster.

That would also mean you don't need a database migration. That's something my idea would have required.

advplyr commented 7 months ago

Yeah exactly. Or we could do /api/media/:mediaType/:mediaId/cover. The mediaType stored on library items is singular so it would be easier on the clients and allow for adding new types without creating new cover routes.

advplyr commented 7 months ago

Another performance improvement was made in v2.6.0 with some api caching thanks to @mikiher here https://github.com/advplyr/audiobookshelf/pull/2343

DDriggs00 commented 7 months ago

It does seem to be slightly faster in v2.6 compared to v2.4/5, but still very noticeably slower than v2.3.

AppUserOne commented 7 months ago

I am running the version: 2.6.0 in Yunohost environment. The peformance (loading the initial library, browsing etc.) is extremely slow as compared to original version a while ago (probably 2.3.3). Additionally, the app crashes consistently on downloading any larger podcasts (500+ espisodes). Is there a way to do a fresh install of version 2.3.3 in Yunohost package while these performance issues/bugs are addressed? The official younohost repository only allows the latest version of be installed. I tested both fixes suggested in this thread (disabling watcher, ignoring pre-fix) with minimal improvements.

JBlond commented 7 months ago

I am running the version: 2.6.0 in Yunohost environment. The peformance (loading the initial library, browsing etc.) is extremely slow as compared to original version a while ago (probably 2.3.3). Additionally, the app crashes consistently on downloading any larger podcasts (500+ espisodes). Is there a way to do a fresh install of version 2.3.3 in Yunohost package while these performance issues/bugs are addressed? The official younohost repository only allows the latest version of be installed. I tested both fixes suggested in this thread (disabling watcher, ignoring pre-fix) with minimal improvements.

Yes, with docker compose https://www.audiobookshelf.org/docs#docker-compose-install you can define the tag. instead of :latest you can use :2.3.3

AppUserOne commented 6 months ago

Yes, with docker compose https://www.audiobookshelf.org/docs#docker-compose-install you can define the tag. instead of :latest you can use :2.3.3

Thanks for the quick response. But I was hoping to find something like the following, supporting install of an earlier version. I believe docker compose method will require separate docker/docker-compose install outside of Yunohost web GUI and I haven't found some good stable guides around that setup https://github.com/YunoHost-Apps/audiobookshelf_ynh

JBlond commented 6 months ago

You can, but you have to modify the install manifest https://github.com/YunoHost-Apps/audiobookshelf_ynh/blob/master/manifest.toml#L39-L41

AppUserOne commented 6 months ago

You can, but you have to modify the install manifest https://github.com/YunoHost-Apps/audiobookshelf_ynh/blob/master/manifest.toml#L39-L41

Thank you so much. I was able to copy the repository over and modify the file to get to install the required version from within Yunohost web GUI. No more lagging loads and performance issues. Now hoping they all get resolved so that I can upgrade to the latest one.

advplyr commented 6 months ago

Another performance update in v2.7.1 for podcasts

advplyr commented 6 months ago

At this point in v2.7.1 the performance issues should mostly be resolved.

For larger libraries it won't be as fast as v2.3.x and below because the data is being fetched as needed from the database instead of all being stored in memory. For obvious reasons we couldn't keep storing the entire database in memory.

There is still more to be done in optimizing the queries and reducing the amount of the data returned from the API but performance improvements never really end.

Let me know what other performance issues you are seeing

ejstacey commented 6 months ago

Just some feedback: 2.7.1 is night and day faster than 2.7.0. I'm finally able to listen to some podcasts I have with near 1000 eps downloaded! I'll still keep an eye out, but it feels like the issue is completely solved for me. Thanks for sticking with this!

Eftimin70 commented 6 months ago

Unfortunately version 2.7.1 did not change much on my side. I'm not using podcasts, just audiobooks and radio plays. On my setup the new version is still a lot slower than 2.3.3.

advplyr commented 6 months ago

@Eftimin70 Roughly what are the size of your libraries? Anything notable like audiobooks with thousands of audio tracks or anything like that you can share?

Are you running the server with Docker desktop on Windows or how are you running the server?

Are you using a network file system for the database?

Eftimin70 commented 6 months ago

@advplyr Here are my stats for the radio plays:

4165 Items in Library 3816 Overall Hours 361 Authors 305.7 Size (GB) 43901 Audio Tracks

Audiobookshelf is running in docker on a qnap with an Intel Celeron J4125. The Folders are directly hosted on the qnap server which is equipped with 20GB of RAM.

kaldigo commented 6 months ago

For me the library loading has been much faster with this update. The only problem I am having is that covers are taking up to 22 sec each to load image image image Thats with a library of 1319 audiobooks.

On the home page the load time is up to 2 sec per image image image image

Audiobookshelf is running via docker on unraid. The server is running a i9-9900k with 32GB of RAM.

hoggatt commented 6 months ago

Just updated from 2.3.3. to 2.7.1 and for podcasts, it is fast! Thanks for the effort! The home page takes a bit to start when the android app is initially loaded (although it did in 2.3.3 as well, maybe a touch slower in the newest version). Web version loads right away. The "Latest" page (which I use often) loads very quick for both the app and the web.

If I go into the library section (again, android app), a podcast with 750 episodes takes a second or two to load. Not bad at all! The page itself shows up all at once (after everything is loaded in the background). My opinion is that loading a skeleton page, then populating the content (youtube does this) makes it seem faster. For some reason it takes a second when I hit the back arrow too. The library section is blazing fast on the web though, so it might be an android app-related issue.

JBlond commented 6 months ago

With the new version, the "re-scan" of a library is much faster than before. Thanks for the effort.