search

advplyr / audiobookshelf

Self-hosted audiobook and podcast server
https://audiobookshelf.org
GNU General Public License v3.0
5.82k stars 396 forks source link

[Bug]: FATAL: [Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error #3123

Closed hardwareadictos closed 4 days ago

hardwareadictos commented 1 week ago

What happened?

When adding a large amount of ebooks to a library (like 30000) container crashes with error ERROR: [Watcher] Error: ENOSPC: System limit for number of file watchers reached on library scan

What did you expect to happen?

Scaning taking long but not crashing

Steps to reproduce the issue

Just adding large amount of ebooks and scanning

Audiobookshelf version

2.10.1

How are you running audiobookshelf?

Docker

What OS is your Audiobookshelf server hosted from?

Linux

If the issue is being seen in the UI, what browsers are you seeing the problem on?

None

Logs

Nothing relevant, just:

[2024-07-05 22:59:32.266] ERROR: [Watcher] Error: ENOSPC: System limit for number of file watchers reached, watch '/path/to/book' (Watcher.js:77)
 For each file once the limit is reached

Additional Notes

No response

advplyr commented 1 week ago

Is it crashing the container or just showing an error? You need to increase the number of file watchers on your system or you can disable the watcher.

hardwareadictos commented 1 week ago

Is it crashing the container or just showing an error? You need to increase the number of file watchers on your system or you can disable the watcher.

It just hungs and reports 502 error on browser.

Just applied this solution, which consists on a little kernel parameter: https://stackoverflow.com/questions/55763428/react-native-error-enospc-system-limit-for-number-of-file-watchers-reached

Will check during a week and report back.

hardwareadictos commented 1 week ago

Well, no more limit errors, but now i can reproduce this on every scan on this library:

AudioBookShelf-PRO  | [2024-07-06 09:52:04.258] FATAL: [Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {
AudioBookShelf-PRO  |   <rejected> Error
AudioBookShelf-PRO  |       at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)
AudioBookShelf-PRO  |       at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50
AudioBookShelf-PRO  |       at new Promise (<anonymous>)
AudioBookShelf-PRO  |       at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)
AudioBookShelf-PRO  |       at /node_modules/sequelize/lib/sequelize.js:315:28
AudioBookShelf-PRO  |       at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)
AudioBookShelf-PRO  |       at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)
AudioBookShelf-PRO  |       at async /node_modules/sequelize/lib/model.js:2541:13
AudioBookShelf-PRO  |       at async Promise.all (index 1)
AudioBookShelf-PRO  |       at async /node_modules/sequelize/lib/model.js:2522:9 {
AudioBookShelf-PRO  |     name: 'SequelizeUniqueConstraintError',
AudioBookShelf-PRO  |     errors: [ [ValidationErrorItem], [ValidationErrorItem] ],
AudioBookShelf-PRO  |     parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
AudioBookShelf-PRO  |       errno: 19,
AudioBookShelf-PRO  |       code: 'SQLITE_CONSTRAINT',
AudioBookShelf-PRO  |       sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
AudioBookShelf-PRO  |     },
AudioBookShelf-PRO  |     original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
AudioBookShelf-PRO  |       errno: 19,
AudioBookShelf-PRO  |       code: 'SQLITE_CONSTRAINT',
AudioBookShelf-PRO  |       sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
AudioBookShelf-PRO  |     },
AudioBookShelf-PRO  |     fields: [ 'bookId', 'seriesId' ],
AudioBookShelf-PRO  |     sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
AudioBookShelf-PRO  |   }
AudioBookShelf-PRO  | } (Server.js:165)

And this does't hung the container, this directly makes it crash

advplyr commented 1 week ago

Can you share your docker config and if you are using remote file systems for the mounts

hardwareadictos commented 1 week ago

Can you share your docker config and if you are using remote file systems for the mounts

Sure!

  audiobookshelf:
    user: 2500:2501
    restart: unless-stopped
    container_name: AudioBookShelf-PRO
    image: ghcr.io/advplyr/audiobookshelf:2.11.0
    ports:
      - 13378:80
    volumes:
      - /mnt/Audiolibros:/audiobooks
      - /mnt/Libros:/libros
      - /mnt/podcasts:/podcasts
      - ./config:/config
      - /mnt/audiobookshelf_metadata:/metadata
    environment:
      - TZ=Europe/Madrid

And yes, audiobooks, podcasts and books are stored on a remote NFS. It has full permissions for the UID/GID provided on the docker-compose file and over a 1G network.

Do you need any more info?

nichwall commented 1 week ago

Are there supposed to be 2 slashes in the audio book mount? Not sure if that's a typo.

hardwareadictos commented 1 week ago

Are there supposed to be 2 slashes in the audio book mount? Not sure if that's a typo.

Yes its a typo, just edited the real mountpoints before posting them here, edited and corrected it-

mikiher commented 6 days ago

From what I read above, it looks like you have watcher enabled and are running (automated?) library scans.

First of all, since your libraries are NFS mounted, I suspect that the watcher is not working for you (which is likely why you need to run library scans). Watcher only works reliably when watched files and directories are on the local file system - when they're mounted on a remote NFS, watcher will appear to be running but will not get notified of file change events.

Therefore, I'd advise disabling watcher (uncheck Settings -> Enable watcher).

Once you do that, please let us know if your library scans are still crashing.

hardwareadictos commented 6 days ago

From what I read above, it looks like you have watcher enabled and are running (automated?) library scans.

First of all, since your libraries are NFS mounted, I suspect that the watcher is not working for you (which is likely why you need to run library scans). Watcher only works reliably when watched files and directories are on the local file system - when they're mounted on a remote NFS, watcher will appear to be running but will not get notified of file change events.

Therefore, I'd advise disabling watcher (uncheck Settings -> Enable watcher).

Once you do that, please let us know if your library scans are still crashing.

Good morning!!

Will try and report soon.

Something to take into account: I have all my libraries using NFS shared storage (podcasts and audiobooks) with watchers enabled working fine. I also have a Jellyfin instance on the same server working with something similar to watchers (if not the same) enabled on my media directories with NFS shared storage also working fine for more than 4 years. The only difference between my other libraries is that ebook ones have more than 130000 files (which maybe is a lot for a sqlite database).

Let me test and report back. Thanks a lot for the support :)

hardwareadictos commented 6 days ago

Well, same issue as before, watchers disabled for ebooks library and scanned manually (logs on debug mode):

AudioBookShelf-PRO  | [2024-07-10 14:15:08.761] DEBUG: Parsing metadata from epub at "/libros/Zygmunt Miloszewski/El caso Telak (36480)/El caso Telak - Zygmunt Miloszewski.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.786] DEBUG: [LibraryScan] "Libros": "El caso Telak (36480)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.794] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/1ca08853-414c-4def-abe2-c2ef787a0e16/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.794] DEBUG: [LibraryScan] "Libros": Library item "Zygmunt Miloszewski/El caso Telak (36480)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.810] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/El policia rural y otros relatos (36192)/El policia rural y otros relatos - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.826] DEBUG: [LibraryScan] "Libros": "El policia rural y otros relatos (36192)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.830] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/9dfc984a-ee01-4df6-b7ec-b89713b37236/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.831] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/El policia rural y otros relatos (36192)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.847] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/La estampida (22473)/La estampida - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.864] DEBUG: [LibraryScan] "Libros": "La estampida (22473)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.870] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/55f25dd3-7e2b-42d7-91fc-6bcff80da5f4/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.871] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/La estampida (22473)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.889] DEBUG: Parsing metadata from epub at "/libros/el Inca Garcilaso de la Vega/Comentarios reales de los Incas (53075)/Comentarios reales de los Incas - el Inca Garcilaso de la Vega.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.935] DEBUG: [LibraryScan] "Libros": "Comentarios reales de los Incas (53075)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.941] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/0656edd9-bc02-4f55-b2c9-a9a018c15e23/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.942] DEBUG: [LibraryScan] "Libros": Library item "el Inca Garcilaso de la Vega/Comentarios reales de los Incas (53075)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.956] DEBUG: Parsing metadata from epub at "/libros/Zoran Zivkovic/Historias imposibles (37178)/Historias imposibles - Zoran Zivkovic.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.973] DEBUG: [LibraryScan] "Libros": "Historias imposibles (37178)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.980] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/99c54493-8e97-4fca-bd6c-240f33b95960/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.981] DEBUG: [LibraryScan] "Libros": Library item "Zoran Zivkovic/Historias imposibles (37178)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:08.996] DEBUG: Parsing metadata from epub at "/libros/Zecharia Sitchin/El final de los tiempos (22201)/El final de los tiempos - Zecharia Sitchin.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.020] DEBUG: [LibraryScan] "Libros": "El final de los tiempos (22201)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.028] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/0fc5a3aa-5146-47b2-9162-d1a12226eae7/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.029] DEBUG: [LibraryScan] "Libros": Library item "Zecharia Sitchin/El final de los tiempos (22201)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.045] DEBUG: Parsing metadata from epub at "/libros/Zoe Valdes/La nada cotidiana (37644)/La nada cotidiana - Zoe Valdes.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.060] DEBUG: [LibraryScan] "Libros": "La nada cotidiana (37644)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.068] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/c8925063-6854-4d77-ab33-6b72ce1e2beb/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.069] DEBUG: [LibraryScan] "Libros": Library item "Zoe Valdes/La nada cotidiana (37644)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.084] DEBUG: Parsing metadata from epub at "/libros/Zoe Valdes/Traficantes de belleza (27334)/Traficantes de belleza - Zoe Valdes.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.099] DEBUG: [LibraryScan] "Libros": "Traficantes de belleza (27334)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.107] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/29fb669f-42ba-4302-8f8c-8ad37037fd8c/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.108] DEBUG: [LibraryScan] "Libros": Library item "Zoe Valdes/Traficantes de belleza (27334)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.123] DEBUG: Parsing metadata from epub at "/libros/diostuitero/Palabra de Dios tuitero (37982)/Palabra de Dios tuitero - diostuitero.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.143] DEBUG: [LibraryScan] "Libros": "Palabra de Dios tuitero (37982)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.148] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/d56acb63-6355-4387-a6be-66f6fdd82ab6/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.149] DEBUG: [LibraryScan] "Libros": Library item "diostuitero/Palabra de Dios tuitero (37982)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.168] DEBUG: Parsing metadata from epub at "/libros/Zhuang Zi/Zhuang Zi (69044)/Zhuang Zi - Zhuang Zi.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.188] DEBUG: [LibraryScan] "Libros": "Zhuang Zi (69044)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.194] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/1acee10c-2507-4fc0-b800-cdf890f51be0/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.195] DEBUG: [LibraryScan] "Libros": Library item "Zhuang Zi/Zhuang Zi (69044)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.209] DEBUG: Parsing metadata from epub at "/libros/Zygmunt Bauman/El arte de la vida (49790)/El arte de la vida - Zygmunt Bauman.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.224] DEBUG: [LibraryScan] "Libros": "El arte de la vida (49790)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.231] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/733481c5-7f96-484b-873c-06895e33dc1e/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.232] DEBUG: [LibraryScan] "Libros": Library item "Zygmunt Bauman/El arte de la vida (49790)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.244] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/La caravana perdida (32027)/La caravana perdida - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.259] DEBUG: [LibraryScan] "Libros": "La caravana perdida (32027)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.265] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/ca9ea948-6f54-4748-b566-3e4d330f4fca/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.266] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/La caravana perdida (32027)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.280] DEBUG: Parsing metadata from epub at "/libros/Zygmunt Bauman/Vigilancia liquida (60408)/Vigilancia liquida - Zygmunt Bauman.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.294] DEBUG: [LibraryScan] "Libros": "Vigilancia liquida (60408)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.299] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/f094d4b6-dfe6-46c4-9800-3de20b3cd4f5/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.300] DEBUG: [LibraryScan] "Libros": Library item "Zygmunt Bauman/Vigilancia liquida (60408)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.315] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/Caravana de heroes (22605)/Caravana de heroes - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.330] DEBUG: [LibraryScan] "Libros": "Caravana de heroes (22605)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.335] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/7ff88f25-8444-4b57-9aa7-616977500210/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.336] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/Caravana de heroes (22605)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.350] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/Meseta negra (27267)/Meseta negra - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.362] DEBUG: [LibraryScan] "Libros": "Meseta negra (27267)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.370] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/a9c8c5c7-a03e-4877-a78e-3a56e32edd1b/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.370] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/Meseta negra (27267)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.386] DEBUG: Parsing metadata from epub at "/libros/Zecharia Sitchin/La escalera al cielo (15921)/La escalera al cielo - Zecharia Sitchin.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.411] DEBUG: [LibraryScan] "Libros": "La escalera al cielo (15921)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.429] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/e1924b78-b877-4c90-ae68-5d3f8a8b04a3/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.430] DEBUG: [LibraryScan] "Libros": Library item "Zecharia Sitchin/La escalera al cielo (15921)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.445] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/Al oeste del Pecos (22108)/Al oeste del Pecos - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.458] DEBUG: [LibraryScan] "Libros": "Al oeste del Pecos (22108)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.463] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/20009c1d-d882-480c-b4c4-ffbcb09f1836/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.464] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/Al oeste del Pecos (22108)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.486] DEBUG: Parsing metadata from epub at "/libros/bell hooks/El feminismo es para todo el mundo (65396)/El feminismo es para todo el mundo - bell hooks.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.505] DEBUG: [LibraryScan] "Libros": "El feminismo es para todo el mundo (65396)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.512] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/5bca3443-2b0c-4cd3-9249-615cda766450/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.513] DEBUG: [LibraryScan] "Libros": Library item "bell hooks/El feminismo es para todo el mundo (65396)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.528] DEBUG: Parsing metadata from epub at "/libros/Zoe Oldenbourg/La piedra angular (58100)/La piedra angular - Zoe Oldenbourg.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.553] DEBUG: [LibraryScan] "Libros": "La piedra angular (58100)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.559] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/207bb837-834d-4372-a280-849b1d8f1843/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.560] DEBUG: [LibraryScan] "Libros": Library item "Zoe Oldenbourg/La piedra angular (58100)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.577] DEBUG: Parsing metadata from epub at "/libros/Zane Grey/El cuchillo fatidico (22626)/El cuchillo fatidico - Zane Grey.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.593] DEBUG: [LibraryScan] "Libros": "El cuchillo fatidico (22626)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.599] DEBUG: [LibraryScan] "Libros": Found metadata file "/metadata/items/0e208e81-8389-4e30-bc6e-c274f0e258f1/metadata.json" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.600] DEBUG: [LibraryScan] "Libros": Library item "Zane Grey/El cuchillo fatidico (22626)" is up-to-date (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.607] DEBUG: Parsing metadata from epub at "/libros/Yvonne Whittal/Cadenas de oro (65639)/Cadenas de oro - Yvonne Whittal.epub" (parseEpubMetadata.js:71)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.623] DEBUG: [LibraryScan] "Libros": "Cadenas de oro (65639)" Getting metadata with precedence [folderStructure, audioMetatags, nfoFile, txtFiles, opfFile, absMetadata] (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.684] DEBUG: [LibraryScan] "Libros": Extracted embedded cover from ebook file at "/metadata/items/3bc3f8c9-1901-4f55-97a9-0aa0034b996d/cover.jpg" for book "Cadenas de oro" (LibraryScan.js:131)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.774] DEBUG: [ApiCacheManager] bookSeries.afterCreate: Clearing cache (ApiCacheManager.js:21)
AudioBookShelf-PRO  | [2024-07-10 14:15:09.785] FATAL: [Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {
AudioBookShelf-PRO  |   <rejected> Error
AudioBookShelf-PRO  |       at Database.<anonymous> (/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27)
AudioBookShelf-PRO  |       at /node_modules/sequelize/lib/dialects/sqlite/query.js:183:50
AudioBookShelf-PRO  |       at new Promise (<anonymous>)
AudioBookShelf-PRO  |       at Query.run (/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12)
AudioBookShelf-PRO  |       at /node_modules/sequelize/lib/sequelize.js:315:28
AudioBookShelf-PRO  |       at async SQLiteQueryInterface.insert (/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)
AudioBookShelf-PRO  |       at async BookSeries.save (/node_modules/sequelize/lib/model.js:2490:35)
AudioBookShelf-PRO  |       at async /node_modules/sequelize/lib/model.js:2541:13
AudioBookShelf-PRO  |       at async Promise.all (index 1)
AudioBookShelf-PRO  |       at async /node_modules/sequelize/lib/model.js:2522:9 {
AudioBookShelf-PRO  |     name: 'SequelizeUniqueConstraintError',
AudioBookShelf-PRO  |     errors: [ [ValidationErrorItem], [ValidationErrorItem] ],
AudioBookShelf-PRO  |     parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
AudioBookShelf-PRO  |       errno: 19,
AudioBookShelf-PRO  |       code: 'SQLITE_CONSTRAINT',
AudioBookShelf-PRO  |       sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
AudioBookShelf-PRO  |     },
AudioBookShelf-PRO  |     original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
AudioBookShelf-PRO  |       errno: 19,
AudioBookShelf-PRO  |       code: 'SQLITE_CONSTRAINT',
AudioBookShelf-PRO  |       sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
AudioBookShelf-PRO  |     },
AudioBookShelf-PRO  |     fields: [ 'bookId', 'seriesId' ],
AudioBookShelf-PRO  |     sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
AudioBookShelf-PRO  |   }
AudioBookShelf-PRO  | } (Server.js:171)
AudioBookShelf-PRO exited with code 0

It scans like 135 books of +- 130000 and it just crashes on this constrain.

mikiher commented 6 days ago

If your watchers are working fine, why would you need an automated library scan? the Audiobookshelf watcher is supposed to detect any file changes in any of the directories of your library, and add books automatically when they're detectedm without requiring a library scan. Is it doing that?

hardwareadictos commented 6 days ago

If your watchers are working fine, why would you need an automated library scan? the Audiobookshelf watcher is supposed to detect any file changes in any of the directories of your library, and add books automatically when they're detectedm without requiring a library scan. Is it doing that?

Maybe i didn't explain myself correctly: Watchers are disabled on that library scan, automated scans are also disabled, i'm just triggering a manual scan. Scan is what's crashing.

The thing ends with that constrain error and reboots de container with an exit 0

mikiher commented 6 days ago

If your watchers are working fine, why would you need an automated library scan? the Audiobookshelf watcher is supposed to detect any file changes in any of the directories of your library, and add books automatically when they're detectedm without requiring a library scan. Is it doing that?

Maybe i didn't explain myself correctly: Watchers are disabled on that library scan, automated scans are also disabled, i'm just triggering a manual scan. Scan is what's crashing.

OK, undetstood. Thanks for turning log level to Debug. I'm looking at the logs.

The thing ends with that constrain error and reboots de container with an exit 0

mikiher commented 6 days ago

As with the other bug, I'd appreciate if you attach the whole server side log (including debug printings from the whole last scan until the crash). This kind of failure might hint at some race condition accessing the database, so I want to understand all that happened (and all that didn't happen) prior to the failure.

hardwareadictos commented 6 days ago

As with the other bug, I'd appreciate if you attach the whole server side log (including debug printings from the whole last scan until the crash). This kind of failure might hint at some race condition accessing the database, so I want to understand all that happened (and all that didn't happen) prior to the failure.

No problem! Will grab them in an hour.

mikiher commented 6 days ago

Just the last day (which contains the last scan you ran manually).

On Wed, Jul 10, 2024 at 8:04 PM hardwareadictos @.***> wrote:

As with the other bug, I'd appreciate if you attach the whole server side log (including debug printings from the whole last scan until the crash). This kind of failure might hint at some race condition accessing the database, so I want to understand all that happened (and all that didn't happen) prior to the failure.

No problem! I have the logs from nearly all July. Do you want all of them?

— Reply to this email directly, view it on GitHub https://github.com/advplyr/audiobookshelf/issues/3123#issuecomment-2221036063, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMDFVVWI7AJLNI6FVNXNLLZLVSSRAVCNFSM6AAAAABKNX5OA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRGAZTMMBWGM . You are receiving this because you commented.Message ID: @.***>

hardwareadictos commented 6 days ago

Here you have it:

2024-07-10.txt

thanks! :D

mikiher commented 5 days ago

One more question: has the library "Libros" been successfully scanned before, or were you never able to fully scan it?

hardwareadictos commented 5 days ago

One more question: has the library "Libros" been successfully scanned before, or were you never able to fully scan it?

Never as it's really big, other libraries like podcasts and Audiolibros are being scanned fine.

I even deleted and recreated Libros library to avoid issues creating it the first time without success

mikiher commented 5 days ago

OK, I believe I understand what is happening here.

I think this is a specific issue with the book /libros/Yvonne Whittal/Cadenas de oro (65639)/Cadenas de oro - Yvonne Whittal.epub that crashes Audiobookshelf - I think it has a duplicate series metadata inside its opf file (somehwhat similar to #2650)

Can you possibly attach this specific epub here so I can examine it?

mikiher commented 5 days ago

Yeah, I was able to repro the crash by duplicating the series metadata lines in a random epub's opf file:

    <meta name="calibre:series" content="Series"/>
    <meta name="calibre:series_index" content="1.0"/>
    <meta name="calibre:series" content="Series"/>
    <meta name="calibre:series_index" content="1.0"/>

Then, scanning this epub I get the same crash:

[2024-07-11 23:11:52.903] DEBUG: [ApiCacheManager] bookSeries.afterCreate: Clearing cache (ApiCacheManager.js:21)
[2024-07-11 23:11:52.906] FATAL: [Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {
  <rejected> Error
      at Database.<anonymous> (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\sqlite\query.js:185:27)
      at F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\sqlite\query.js:183:50
      at new Promise (<anonymous>)
      at Query.run (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\sqlite\query.js:183:12)
      at F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\sequelize.js:315:28
      at async SQLiteQueryInterface.insert (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\abstract\query-interface.js:308:21)
      at async BookSeries.save (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\model.js:2490:35)
      at async F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\model.js:2541:13
      at async Promise.all (index 1)
      at async F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\model.js:2522:9 {
    name: 'SequelizeUniqueConstraintError',
    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],
    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
      errno: 19,
      code: 'SQLITE_CONSTRAINT',
      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
    },
    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
      errno: 19,
      code: 'SQLITE_CONSTRAINT',
      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
    },
    fields: [ 'bookId', 'seriesId' ],
    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
  }
} (Server.js:171)
hardwareadictos commented 4 days ago

Yeah, I was able to repro the crash by duplicating the series metadata lines in a random epub's opf file:

    <meta name="calibre:series" content="Series"/>
    <meta name="calibre:series_index" content="1.0"/>
    <meta name="calibre:series" content="Series"/>
    <meta name="calibre:series_index" content="1.0"/>

Then, scanning this epub I get the same crash:

[2024-07-11 23:11:52.903] DEBUG: [ApiCacheManager] bookSeries.afterCreate: Clearing cache (ApiCacheManager.js:21)
[2024-07-11 23:11:52.906] FATAL: [Server] Unhandled rejection: SequelizeUniqueConstraintError: Validation error, promise: Promise {
  <rejected> Error
      at Database.<anonymous> (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\sqlite\query.js:185:27)
      at F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\sqlite\query.js:183:50
      at new Promise (<anonymous>)
      at Query.run (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\sqlite\query.js:183:12)
      at F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\sequelize.js:315:28
      at async SQLiteQueryInterface.insert (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\dialects\abstract\query-interface.js:308:21)
      at async BookSeries.save (F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\model.js:2490:35)
      at async F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\model.js:2541:13
      at async Promise.all (index 1)
      at async F:\VS Code Worksapces\audiobookshelf-local\audiobookshelf\node_modules\sequelize\lib\model.js:2522:9 {
    name: 'SequelizeUniqueConstraintError',
    errors: [ [ValidationErrorItem], [ValidationErrorItem] ],
    parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
      errno: 19,
      code: 'SQLITE_CONSTRAINT',
      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
    },
    original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookSeries.bookId, bookSeries.seriesId] {
      errno: 19,
      code: 'SQLITE_CONSTRAINT',
      sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
    },
    fields: [ 'bookId', 'seriesId' ],
    sql: 'INSERT INTO `bookSeries` (`id`,`sequence`,`createdAt`,`bookId`,`seriesId`) VALUES ($1,$2,$3,$4,$5);'
  }
} (Server.js:171)

Nice found!! Let me check it later, I will exclude this ebook from the library folder later and will reescan to confirm. Thanks for the effort.

hardwareadictos commented 4 days ago

Okay i found the duplicates also and i deleted them with calibre editor:

image

And boila, now it's scanning beyond the book n135 which was the culprit. Now i have another 129750 books to scan and "correct" if it founds another one broken.

From my ignorance: is it posible on the app code side to skip repeated metadata tags and only keep the first one found? Don't know if it's posible, i would say yes because my ereaders can read this book with those repeated tags, but i don't know if that would compromise the app code :)

Edit: I found another one (which was posible):

image

mikiher commented 4 days ago

Of course, something like this should never cause a crash. I'm working on a fix.

hardwareadictos commented 4 days ago

Of course, something like this should never cause a crash. I'm working on a fix.

Nice!! Then im goint to stop "fixing" the books jeje.

Let me know when you find something, will gladly test :)

mikiher commented 4 days ago

The fix #3152 is in review.

hardwareadictos commented 4 days ago

Nice!! Thanks a lot for the support, quick and effective, loving the app, waiting anxiously for the next release 👏👏👏