search

advplyr / audiobookshelf

Self-hosted audiobook and podcast server
https://audiobookshelf.org
GNU General Public License v3.0
6.37k stars 450 forks source link

[Bug]: Restoring from large AudioBookShelf backup results in broken Author images and "Internal Server Error" #2904

Closed ZLoth closed 2 months ago

ZLoth commented 5 months ago

Describe the issue

Because of a botched upgrade of the TrueCharts docker container, I ended up killing my original AudioBookShelf instance and creating a new instance. I then restored the backup from the latest nightly AudioBookShelf backup from Settings → Backup. While my book images and data was restored properly, most of the Author images were broken. In checking the directory /metadata/authors, I did not see the picture uploaded. I ended up deleting the broken image and then doing a quick match to restore the image.

Example screenshot: image

Here is a recording of the issue: https://github.com/advplyr/audiobookshelf/assets/6700159/68e47f52-48f9-4398-817a-ac95a7a1b273

The UUID attached to the author is adeeccd0-cb1a-4647-9516-f8a63fdb24be on my instance, but when I check the /metadata/authors directory, the image does not exist: image

Steps to reproduce the issue

  1. Make a backup from Settings → Backup
  2. Configure a new Docker instance and establish book mapping.
  3. Restore the backup into a new instance.

WHAT SHOULD HAPPEN: Author metadata and images are restored as well as book images and metadata. WHAT REALLY HAPPENS: Author images are not restored.

WORKAROUND 1: Individually remove the image, then do a quick match again to restore data. WORKAROUND 2: Extract the photos from the backup in metadata-authors, then manually copy the photos into metadata/authors.

Audiobookshelf version

2.9.0, 2.11.0

How are you running audiobookshelf?

Docker

Mega-Cookie commented 3 months ago

Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup.

I'm also running a Docker container

MY WORKARROUND:

  1. mkdir authors/ in metadata/
  2. extract metadata-authors/ from the backup file (with WinRAR in my case)
  3. copy contents of metadata-authors/ over to metadata/authors/
ZLoth commented 2 months ago

Had a similar Issue where the metadata/authors/ directory was not created at all after restoring from backup.

Issue is still occurring in 2.11 . But, I was able to restore the authors per your workaround.

nichwall commented 2 months ago

Are you able to get any logs of the backup restore? I'm not able to reproduce the issue of author images not being restored using Docker version 2.11.0

My debug steps:

I also tried:

ZLoth commented 2 months ago

@nichwall : My apologies for the delay as I was not able to attempt a reproduction of the issue until this weekend. My configuration is that I'm running TrueNAS Scale running Dragonfish-24.04.2, and I'm using a Docker container. My testing is that I set up a second test instance of ABS, with the only difference is that my production Config Storage and Metadata Storage, configured to use Host Path, pointed to /mnt/pool/app-config/abs-test instead of my production directory of /mnt/pool/app-config/audiobookshelf , and the port used was different from production.

STEPS TO REPLICATE:

  1. Created a second ABS instance with the name of audiobookshelf-test.
  2. Created a login account.
  3. Went into Settings → Backup.
  4. Restored from my backup. During the import, the socket got disconnected and reconnected.
  5. Refreshed my screen and re-logged in.
  6. Went into Authors.

WHAT SHOULD HAPPEN: Pictures should have been restored. WHAT REALLY HAPPENS: Multiple errors thrown in the logs, and no pictures in the authors directory.

Here is a recording of the issue. My apologies for the big yellow box, as I don't want my users to be displayed. https://github.com/user-attachments/assets/4a7dc434-2f7a-4a5c-a457-bf7b9c7dcc4e

Here are the log files: Logs.zip

nichwall commented 2 months ago

Thanks for providing more information. I just tried following your steps (using the same hardware, but an entirely new directory path/container). I am not seeing the socket disconnect and reconnect or any errors in my logs. Granted, the largest backup file I have on hand for this is only about 200 MB.

I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try? Based on the log file showing that /metadata/authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg cannot be found while restoring the backup, I'm wondering if it could be:

Can you answer/do the following:

  1. How many author images are in the backup?
  2. How big/what percentage of the backup size is author images?
  3. How big/what percentage of the backup size is from metadata-items?
  4. Verify that metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg exists in the backup.
  5. Look at /metadata/authors after the extraction and see if any files exist in the file system?
ZLoth commented 2 months ago

I noticed in your video that your backup is 1.4 GB, do you have any smaller backups you could try?

Negative. This is my daily backup of my production instance, and has over 4,400 books (thanks Plus catalog)

  1. How many author images are in the backup?

image

  1. How big/what percentage of the backup size is author images?
  2. How big/what percentage of the backup size is from metadata-items?

image

  1. Verify that metadata-authors/0003ec65-73d1-4c34-bcd2-4ba65abb6d7d.jpg exists in the backup.

image

  1. Look at /metadata/authors after the extraction and see if any files exist in the file system?

As you can see from the following five-minute recording where I went from adding the app in TrueNAS scale as a second instance, setting up the paths, restoring the backup, etc , both /metadata/items and /metadata/authors do not exist until I attempt restore from backup. After restoration from backup, items exist, but authors does not exist. It's only when I update the photo does the authors directory exist.

https://github.com/user-attachments/assets/410d5c42-9287-41af-baa1-09b13c03b5d5

Log files: Logs.zip

nichwall commented 2 months ago

Why is absdatabase.sqlite in your metadata folder (at 2:27)? Having config and metadata at the same mount point may be causing some of the problems.

ZLoth commented 2 months ago

Here is a recording where I created the subdirectories config and metadata as subdirectories under /mnt/pool/app-config/abs-test:

https://github.com/user-attachments/assets/3728bc04-e08a-471d-9da8-1fc1d699f962

Here is a recording where I created directories /mnt/pool/app-config/abs-config and /mnt/pool/app-config/abs-metadata:

https://github.com/user-attachments/assets/de0d6df3-ca9f-4cb7-9ac2-2febd59342e3

Still same issue.

ZLoth commented 2 months ago

And, just for fun, I'm using the IX defaults for the metadata and config:

https://github.com/user-attachments/assets/7fc44168-7ace-4862-823e-8dc4b00a82a5

ZLoth commented 2 months ago

What is the order of the restoration of the backup? Is it items, then the sqlite DB, then authors?

nichwall commented 2 months ago

SQLite, items, then authors https://github.com/advplyr/audiobookshelf/blob/master/server%2Fmanagers%2FBackupManager.js#L171-L233

nichwall commented 2 months ago

If you're willing to DM a download link for the backup file, I can try and recreate the issue specifically from your backup. (Though, maybe change the admin user password to "password" or something so it's easier for me to actually log in and verify things)

ZLoth commented 2 months ago

@nichwall : As the file is 1.5 GB in size, a WeTransfer link has been sent to you directly. The link will expire on July 24th.

nichwall commented 2 months ago

Thanks, I was able to reproduce the server crash during restoring the backup with your backup file and was able to log in using the new password you created. I'll see if I can find more information.

nichwall commented 2 months ago

Okay, pretty sure I found the problem.

In your /metadata/authors folder, there are only files (no directories). Backups from my main server (running on Synology) have a hidden folder @eaDir in /metadata/authors. It looks like folders are only created during extraction if there is a folder in the path, so when metadata-authors/ is extracted from the backup, you don't have a folder under metadata-authors/, so the authors directory is not created in the file system. This does not matter for the database because it is extracting a single file to an existing folder, and the metadata-items/ part of the backup is a bunch of folders so /metadata/items gets created.

I added a dummy folder (tried different names to make sure alphabetical sorting didn't matter) to your /metadata/authors folder, made a backup, and then was able to successfully restore the backup on a new server. I also verified that deleting the hidden folder from my backup caused restoring from backup to fail.

ZLoth commented 2 months ago

As a test, I manually created the authors and items folders, chown-ed them to be the apps owner, then performed the restore. This time, no crash, and the authors were restored. So, per your pull request, ensuring that authors and Items folders are created prior to restore is a good fix.

https://github.com/user-attachments/assets/54cb0b9b-980f-4dc8-9703-6d5311e5a990

And all of the authors were restored.

https://github.com/user-attachments/assets/0b2b7995-103e-4f19-b1c9-a8738fff7d3f

Thank you for tracking this issue down. I look forward to the fix, although hopefully, I won't have to do a restore for quite a while.

ZLoth commented 2 months ago

Thank you!

github-actions[bot] commented 2 months ago

Fixed in v2.12.0.