search

DobyTang / LazyLibrarian

This project isn't finished yet. Goal is to create a SickBeard, CouchPotato, Headphones-like application for ebooks. Headphones is used as a base, so there are still a lot of references to it.
732 stars 70 forks source link

calibredb failed, Text file busy #923

Closed ethanopp closed 7 years ago

ethanopp commented 7 years ago

docker linux, QNAP

Current Version: 4f2f8857678ab423cd2831e6a6924b89ec5c5a7a

I am getting the following error when trying to use the calibredb import program: calibredb failed, Text file busy

Book dir and calibre metadata.db both have 777...

Log snippet:

2017-07-19 20:49:31 WARNING Residual files remain in /downloads/completed/LazyLibrarian/Ashlee Vance - Elon Musk Tesla, SpaceX, and the Quest for a Fantastic Future (PDF&EPUB&MOBİ).fail
2017-07-19 20:49:31 ERROR Postprocessing for Elon Musk - Ashlee Vance has failed: calibredb failed, Text file busy
philborman commented 7 years ago

I did a quick google for "text file busy" and it seems to crop up for all sorts of reasons. Doesn't necessarily mean a permissions issue, you can sometimes get that error message if unable to delete/rename a file, some other program using a file, etc. Unfortunately the small debug log snipped doesn't give enough info to go on. Can you post larger segment showing the whole postprocessor run on that book (or use the new log creator on lazylibrarian log page). Does it only happen on the one book? What version of calibredb in case that's relevant. I use calibredb myself when importing and haven't seen this message, so don't really know where to start looking. I was on calibre 2.75 until a few days ago, just switched to calibre 3.4 but haven't tested that much yet.

ethanopp commented 7 years ago

Just updated calibre to 3.4 and LL to 242dea93c74f16c2c3149d22cae6a817fa962cf5

Same issue... Sorry for not attaching the firs time, I was going to post but of course as soon as tried to replicate manually none of the books would download... Waited for one to finally get grabbed so I could grab just the post-processor log (attached now)

The first attempt was on a book that LL downloaded multiple file types of so it went into a directory, this second attempt only downloaded the single .epub so there was no directory created for it, but same issue both times with different books...

debug.zip

UPDATE: Doesnt seem like the save debug log worked as expected... Pasted here as well:

2017-07-20 09:25:51 INFO Debug log saved as /config/log/debug.zip
2017-07-20 09:25:51 DEBUG 20 lines written to /config/log/debug.log
2017-07-20 09:25:51 DEBUG Redacted 0 passwords/apikeys
2017-07-20 09:22:31 INFO Debug log saved as /config/log/debug.zip
2017-07-20 09:22:31 DEBUG 20 lines written to /config/log/debug.log
2017-07-20 09:22:31 DEBUG Redacted 0 passwords/apikeys
2017-07-20 09:22:11 DEBUG Restart processDir job in 10 minutes
2017-07-20 09:22:11 DEBUG Stop processDir job
2017-07-20 09:22:11 INFO 0 books/mags processed.
2017-07-20 09:22:11 DEBUG Closest match (30%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:22:11 DEBUG Snatched torznab Walter Isaacson - Steve Jobs LL.(11084145) is not in download directory
2017-07-20 09:22:11 INFO Looking for eBook Walter Isaacson - Steve Jobs LL.(11084145) in /downloads/completed/LazyLibrarian
2017-07-20 09:22:11 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:11 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:06 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:06 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:01 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:01 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:01 DEBUG Walter Isaacson - Steve Jobs LL.(11084145) was sent to TRANSMISSION
2017-07-20 09:22:01 DEBUG Closest match (30%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:22:01 DEBUG Snatched torznab Gone Girl - Gillian Flynn is not in download directory
2017-07-20 09:22:01 INFO Looking for eBook Gone Girl - Gillian Flynn in /downloads/completed/LazyLibrarian
2017-07-20 09:22:00 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:00 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:00 DEBUG Gone Girl - Gillian Flynn was sent to TRANSMISSION
2017-07-20 09:22:00 DEBUG Closest match (33%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:22:00 DEBUG Snatched torznab Robert Galbraith - The Cuckoos Calling LL.(16160834) is not in download directory
2017-07-20 09:22:00 INFO Looking for eBook Robert Galbraith - The Cuckoos Calling LL.(16160834) in /downloads/completed/LazyLibrarian
2017-07-20 09:22:00 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:22:00 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:55 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:55 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:50 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:50 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:50 DEBUG Robert Galbraith - The Cuckoos Calling LL.(16160834) was sent to TRANSMISSION
2017-07-20 09:21:50 DEBUG Closest match (35%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:50 DEBUG Snatched torznab The Chaperone by Laura Moriarty is not in download directory
2017-07-20 09:21:50 INFO Looking for eBook The Chaperone by Laura Moriarty in /downloads/completed/LazyLibrarian
2017-07-20 09:21:50 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:50 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:50 DEBUG The Chaperone by Laura Moriarty was sent to TRANSMISSION
2017-07-20 09:21:50 DEBUG Closest match (41%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:50 DEBUG Snatched torznab Amy Tan - The Valley of Amazement LL.(17383934) is not in download directory
2017-07-20 09:21:50 INFO Looking for eBook Amy Tan - The Valley of Amazement LL.(17383934) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:50 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:50 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:45 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:45 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:40 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:40 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:40 DEBUG Amy Tan - The Valley of Amazement LL.(17383934) was sent to TRANSMISSION
2017-07-20 09:21:40 DEBUG Closest match (28%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:40 DEBUG Snatched torznab Elizabeth Smart - My Story LL.(17287028) is not in download directory
2017-07-20 09:21:40 INFO Looking for eBook Elizabeth Smart - My Story LL.(17287028) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:40 DEBUG getTorrentFolder: No response from transmission
2017-07-20 09:21:40 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:40 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:40 DEBUG Elizabeth Smart - My Story LL.(17287028) was sent to TRANSMISSION
2017-07-20 09:21:40 DEBUG Closest match (35%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:40 DEBUG Snatched torrent Jeannette Walls - The Glass Castle LL.(7445) is not in download directory
2017-07-20 09:21:40 INFO Looking for eBook Jeannette Walls - The Glass Castle LL.(7445) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:40 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:40 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:35 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:35 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:30 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:30 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:30 DEBUG Jeannette Walls - The Glass Castle LL.(7445) was sent to TRANSMISSION
2017-07-20 09:21:30 DEBUG Closest match (23%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:30 DEBUG Snatched torznab Kristin Hannah - Between Sisters LL.(217432) is not in download directory
2017-07-20 09:21:30 INFO Looking for eBook Kristin Hannah - Between Sisters LL.(217432) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:30 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:30 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:25 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:25 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:20 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:20 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:20 DEBUG Kristin Hannah - Between Sisters LL.(217432) was sent to TRANSMISSION
2017-07-20 09:21:20 DEBUG Closest match (35%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:20 DEBUG Snatched torznab Revolution - Jennifer Donnelly.epub is not in download directory
2017-07-20 09:21:20 INFO Looking for eBook Revolution - Jennifer Donnelly.epub in /downloads/completed/LazyLibrarian
2017-07-20 09:21:20 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:20 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:20 DEBUG Revolution - Jennifer Donnelly.epub was sent to TRANSMISSION
2017-07-20 09:21:20 DEBUG Closest match (17%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:20 DEBUG Snatched torznab Tracey Martin - Another Little Piece of My Heart LL.(18336825) is not in download directory
2017-07-20 09:21:20 INFO Looking for eBook Tracey Martin - Another Little Piece of My Heart LL.(18336825) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:20 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:20 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:15 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:15 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:10 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:10 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:10 DEBUG Tracey Martin - Another Little Piece of My Heart LL.(18336825) was sent to TRANSMISSION
2017-07-20 09:21:10 WARNING Residual files remain in /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.fail
2017-07-20 09:21:10 ERROR Postprocessing for The Girl with the Lower Back Tattoo - Amy Schumer has failed: calibredb failed, Text file busy
2017-07-20 09:21:10 DEBUG [u'/books/metadata.db', 'add', '-1', u'--with-library=/books', u'/downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer']
2017-07-20 09:21:10 DEBUG Importing The Girl with the Lower Back Tattoo - Amy Schumer into calibre library
2017-07-20 09:21:10 DEBUG One format import, best match = epub
2017-07-20 09:21:10 DEBUG Processing eBook 29405093
2017-07-20 09:21:10 DEBUG Found match (100%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer for eBook Amy Schumer - The Girl with the Lower Back Tattoo LL.(29405093)
2017-07-20 09:21:10 DEBUG Found folder (100%) [/downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer] for eBook The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:10 INFO Looking for eBook The Girl with the Lower Back Tattoo - Amy Schumer.epub in /downloads/completed/LazyLibrarian
2017-07-20 09:21:10 DEBUG TRANSMISSION Changing [Amy Schumer - The Girl with the Lower Back Tattoo LL.(29405093)] to [The Girl with the Lower Back Tattoo - Amy Schumer.epub]
2017-07-20 09:21:10 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:10 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:10 DEBUG Amy Schumer - The Girl with the Lower Back Tattoo LL.(29405093) was sent to TRANSMISSION
2017-07-20 09:21:10 DEBUG Closest match (26%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:10 DEBUG Snatched torznab Tarryn Fisher - Marrow LL.(25379213) is not in download directory
2017-07-20 09:21:10 INFO Looking for eBook Tarryn Fisher - Marrow LL.(25379213) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:10 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:10 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:05 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:05 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:00 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:00 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:00 DEBUG Tarryn Fisher - Marrow LL.(25379213) was sent to TRANSMISSION
2017-07-20 09:21:00 DEBUG Closest match (37%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:21:00 DEBUG Snatched torznab Louise Penny - A Trick of the Light LL.(10680329) is not in download directory
2017-07-20 09:21:00 INFO Looking for eBook Louise Penny - A Trick of the Light LL.(10680329) in /downloads/completed/LazyLibrarian
2017-07-20 09:21:00 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:21:00 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:55 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:55 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:50 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:50 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:50 DEBUG Louise Penny - A Trick of the Light LL.(10680329) was sent to TRANSMISSION
2017-07-20 09:20:50 DEBUG Closest match (38%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:20:50 DEBUG Snatched torznab Dale Carnegie - How to Win Friends and Influence People LL.(4865) is not in download directory
2017-07-20 09:20:50 INFO Looking for eBook Dale Carnegie - How to Win Friends and Influence People LL.(4865) in /downloads/completed/LazyLibrarian
2017-07-20 09:20:50 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:50 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:45 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:45 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG Dale Carnegie - How to Win Friends and Influence People LL.(4865) was sent to TRANSMISSION
2017-07-20 09:20:40 DEBUG Closest match (12%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:20:40 DEBUG Snatched torrent TheOmnivoresDilemma is not in download directory
2017-07-20 09:20:40 INFO Looking for eBook TheOmnivoresDilemma in /downloads/completed/LazyLibrarian
2017-07-20 09:20:40 DEBUG getTorrentFolder: No response from transmission
2017-07-20 09:20:40 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG TheOmnivoresDilemma was sent to TRANSMISSION
2017-07-20 09:20:40 DEBUG Closest match (40%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub
2017-07-20 09:20:40 DEBUG Snatched torznab Jane_Harper-The_Dry-ABOOK-WEB-2017-PROLOG is not in download directory
2017-07-20 09:20:40 INFO Looking for eBook Jane_Harper-The_Dry-ABOOK-WEB-2017-PROLOG in /downloads/completed/LazyLibrarian
2017-07-20 09:20:40 DEBUG getTorrentFolder: No response from transmission
2017-07-20 09:20:40 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-20 09:20:40 DEBUG Jane_Harper-The_Dry-ABOOK-WEB-2017-PROLOG was sent to TRANSMISSION
2017-07-20 09:20:40 DEBUG Found 16 files marked "Snatched"
2017-07-20 09:20:40 DEBUG Checking 1 file in /downloads/completed/LazyLibrarian
2017-07-20 09:20:37 INFO Log cleared, level set to [2]- Log Directory is [/config/log]
philborman commented 7 years ago

Ah - that's it. The entry in lazylibrarian config is wrong. config->processing->Calibredb import program: should be the program, not the database. Calibre has a command line program shipped with it called "calibredb" which is what we need to pass to lazylibrarian. It might be that the program is in your system path, in which case just put calibredb in the box, or you may need to put the full path, eg /usr/bin/calibredb. Depends on how docker is set up. If you have access to your docker via ssh the command "which calibredb" will tell you where it is.

ethanopp commented 7 years ago

For reference, I am using the following docker for calibre: https://github.com/aptalca/docker-rdp-calibre

'which calibre' gives me the following: /usr/bin/calibre... went into the dir just to verify and found /usr/bin/calibredb

Updated the config in LL, restarted both calibre and LL, still getting same issue...

Then realized it is because both are in their own containers, so LL cannot see the the /usr/bin of calibre...

Whats the best way to allow LL docker to see the calibredb inside the calibre docker's /usr/bin?

UPDATE: just typing 'calibredb' seems to run the program fine in the calibre docker - still not sure how to get that over to LL though...

Also quick side question while I am going through all of this trouble - does the calibredb import program import the metadata (.opf, covers, etc.) that LL downloads? ...Kind of why I am going through all of this :D

philborman commented 7 years ago

Sorry, I don't use docker. There is another lazylibrarian docker at Linuxserver.io link at the bottom of lazylibrarian git page. Maybe that includes calibredb? Have you tried putting calibredb into your /books folder as lazylibrarian can see that?

ethanopp commented 7 years ago

That's actually the docker I am using - it seems that this may be more of a limitation of calibredb though so I may be going through all of this for nothing.

LL downloads .opf metadata with the books, and I was looking for a way to automatically import that into calibre... I was asking on the calibre IRC though and it seems they are saying that calibredb import program only imports the book files?

Am I getting some incorrect information here? Don't really see the value in LL downloading the metadata if there is no way to import it into calibre (at least for my setup) because calibre is going to have to redownload it all itself from within calibre.

philborman commented 7 years ago

Some book downloads already include the opf file and a cover, some don't. Lazylibrarian will create an opf file for those books that don't have one, but does not modify embedded book metadata or existing opf files. Whether calibre uses it or not depends on calibre settings, there is a lot of conflicting advice out there, but there is a calibre setting called something like "read metadata from file contents" . If this is on, calibre will read the book metadata for as much info as it can get, and then read any additional metadata from the opf file. It seems embedded book metadata takes preference over opf file, so if calibre got all it wanted from the book it won't read the opf.

If the "read metadata from files" is not set, calibre tries to work out the author and title from the filename and ignores the book metadata and opf file.

In lazylibrarian we use calibredb to send the book, cover, opf and then also force calibre to set the suthor and book title to what we expect, as calibre will use embedded data which might not match what we want.

With magazines, which are usually pdf files, there is no useful metadata and calibre happily uses our opf files.

ethanopp commented 7 years ago

Alright.. Thank you for all of this info - it has been extremely helpful!

I think for the most part I have been able to get to the end result I need with the use of calibre's auto add and LL's ability to copy everything into that dir.

I have 1 last small issue...

When LL finds a download with only 1 book it downloads only that file to the download dir - and does not create a sub directory: /downloads/bookname.epub

Once the post processing runs, the sub-dir gets set up correctly, downloads the metadata and copies the book there, but it keeps the original book file in the main download dir for seeding: /downloads/bookname.epub /downloads/book name/bookname.epub /downloads/book name/bookname.opf /downloads/book name/bookname.jpg

Once seeding is complete, rather then deleting the entire sub directory with all of the files, it only deletes the single book file from the main download dir, and I am left with duplicate files from the sub dir, as the auto add feature has copied this entire sub dir already to my calibre dir.

Is there a way to have LL do a "move" instead of "copy" for the auto add feature?

NOTE: I have "Keep original files" unchecked and "Import one format" checked

philborman commented 7 years ago

Just added some code to allow move rather than copy to autoadd if "keep original files" is unchecked. Not sure what this will do to seeding torrents though...

ethanopp commented 7 years ago

Okay I'll give it a shot..

Just a side-note, I think it may need to only apply when a single file is downloaded, because if LL makes a download with more than 1 file, I believe it automatically sets up that sub-dir in the downloads directory.

philborman commented 7 years ago

It depends on the uploader. Lots of torrents and almost all nzb contain a folder with the file(s) in. Problem is if we move the downloaded file(s), what happens if you want to keep seeding? The torrent programs will complain, I think.

ethanopp commented 7 years ago

Maybe instead of move/copy we can use a Hardlink (like sonarr)?

IS there a setting to test out the 'move' for now or is it automatically doing move if "keep original files" is unchecked?

Updated to 9b5daf82e4f1a54fa192f671e79acb42e4330f7e and still seems to be copying

philborman commented 7 years ago

Actually I have just re-read your earlier message and I think I misremembered how lazylibrarian currently works. I had a quick look at the code, it seems this is what happens...

When LL finds a download with only 1 book it downloads only that file to the download dir - and does >not create a sub directory: /downloads/bookname.epub This is the single file downloaded, no directory provided by the uploader

Once the post processing runs, the sub-dir gets set up correctly, downloads the metadata and copies >the book there, but it keeps the original book file in the main download dir for seeding: /downloads/bookname.epub /downloads/book name/bookname.epub /downloads/book name/bookname.opf /downloads/book name/bookname.jpg This subdir is created by lazylibrarian, and the opf and jpg are also created by lazylibrarian

I think if the download was a torrent/magnet the original single file should be kept for seeding. The lazylibrarian created subdir and all it's folders should be moved into autoadd, not copied. Current version does this.

Hard links are a good idea, but I don't know if they work on windows/mac, and across filesystems?

ethanopp commented 7 years ago

I dont think the current version is doing this correctly - or at least its not on mine.

LL is correctly keeping the single file for seeding, and when its done in transmission it deletes that single file from the main download dir, however the sub-dir that is created with the book file that is then moved is not actually being moved - its still being copied.

After all seeding is complete, I have the book in my calibre autoadd dir, but still have that sub dir in my download dir with the book file inside of it.

In regards to syslinks, I dont think they work across filesystems, although each OS is capable (not 100% sure on this one though)

UPDATE: A few more single-file books download last night and LL is still not doing a copy instead of move.

ethanopp commented 7 years ago

Would using the same dir for "Book Library Folder" as my calibre library be what could be causing this issue?

/books is my main dir that has all of the calibre generated folders with random numbers on book titles in it "i.e. /books/Stephen King/Book Name (XX)"

Right now I have the following settings: "Book Library Folder" /books "eBook Foldername Pattern" /books "eBook Filename Pattern" $Title - $Author "calibredb import program" blank "Calibre Auto Add Directory" blank

Since Calibre generates those numbers at the end of the folder names I left the Foldername Pattern as /books... It seems to still be able to pick up where the books are in these sub directories as downloading a book from the LL browser works by clicking on "Open".

When post processing happens, what exactly does LL try to do? Right now it is both creating a sub-dir in my downloads folder (in addition to the file that is still seeding) as well as placing a copy in the /books folder, even when I have the Calibre Auto Add Directory setting blank.

I have a feeling it is trying to place the sub dir that it creates in the download dir into the /book dir and my settings above may be throwing it off... not sure if that is exactly how the post process works though.

Thanks

philborman commented 7 years ago

Sharing the calibre library folder is fine, but works best if you use calibredb for the reason you have noticed, calibre adds a number to the end of the book foldername. It uses that as it's own internal identifier, and you only get to know the number if you use calibredb. With calibredb we get to know exactly where calibre put the book so we can link to it. Kovid (calibre author) has been asked to change this many times, but he won't. There is really no need to put the id number in the foldername, it's already in the calibre database and the calibre generated opf files, and modifying the foldername just makes it difficult for 3rd party programs. There is a similar issue with renaming authors or book titles in the calibre program, all the folder names move too.

If not using calibredb, auto-add is the next best option. LazyLibrarian will postprocess into a temporary directory, and then scan the calibre library looking for author directory, and scan that looking for any new books.

When postprocessing, if the downloaded torrent/nzb is a single file lazylibrarian will first create a sub-folder using the filename, and copy the book into that directory. This is so that we leave the seeding copy alone, and if we need to create opf and cover images they are kept in a folder with the book.

If you are not using calibre auto-add or calibredb then lazylibrarian will copy/move this new folder (the one we created or the downloaded folder) into the lazylibrarian library using the eBook folder name pattern. Usually author_dir with each book it it's own sub-dir. The assumption is that if you are not using calibre auto-add and not using calibredb, you aren't using calibre at all, so we move it ourselves using our own naming scheme. We move all configured book formats, plus opf, plus jpg, renaming them so they all start with the book title.

If you use /books for both calibre and lazylibrarian, using the default naming pattern for lazylibrarian ebook folder will mean you end up with some book folders with calibre numbers, some without. Those with a number are the only ones calibre knows about, lazylibrarian knows about both. When you next update your calibre library, it will import the ones without a number, and rename their folders adding its own number to the end. After that lazylibrarian can't find them, but if you use lazylibrarian libraryscan it will rescan the calibre library and find them again including the number. Not perfect, but that's why calibredb is the preferred option.

ethanopp commented 7 years ago

Ah okay thanks for laying that out... an idea:

Would it be possible to set up an option for when using Auto-Add to just move the .epub file that is downloaded to a specific dir as opposed to creating the sub dir? ...And if possible ignore the .jpg and .opf's?

It seems like really all Calibre needs is the book file placed in its dir, and with auto-add turned on in calibre, it can automatically download the cover/metadata and set up the proper directories.

Workflow would look something like this:

  1. LL downloads into a download dir
  2. Duplicate is made (As to not screw up seeding file)
  3. Duplicate file is moved to specified dir
  4. The specified dir is enabled on the calibre side for the auto-add, calibre detects the file and automatically sets up all of its folders while downloading meta/cover.

Right now it seems like the subdir is getting created in the download folder, everything is getting copied over to calibre autoadd dir, and then calibre just detects the book file to import, so its sets up all of its dir's and re-downloads the cover and metadata because it doesn't pull those in from the auto-add feature by just placing those files in the directory (Doesnt read .opf/.jpg).

I've tried the calibredb approach but its rather complex since both LL and Calibre are running in their own docker containers.

philborman commented 7 years ago

I think auto-add should move files if "keep original files" is not set, would have to check. We include the opf and jpg in the auto add as if calibre is set to import using file contents it can use them. Another thought, can you copy calibredb into the /books folder, since lazylibrarian can see that? You should be able to put /books/calibredb as your calibredb path in lazylibrarian config. Actually I just had a quick look, it's just a wrapper so you would need the whole calibre installation.

ethanopp commented 7 years ago

"Keep original files" currently is unchecked and it's still doing the copy instead of move.

The .opf/.jpg seem to be getting copied over but then once in calibre dir it looks like it's only doing a "move" on the epub, because I have the opfs/jpgs sitting in the main dir as well as in the sub dirs calibre creates - however the epub gets moved correctly and is not duplicated in the auto add dir

The main concern with copying calibredb into LL is that I do not think it will run correctly. Each program is in its own docker (virtual environment) so the actually calibre program doesn't exist in LL world and LL doesn't exist in calibre world... Will try playing with this further as I may be able to get to the desired result by linking the containers - ideally I'd prefer auto add approach though because linking the 2 would force me to make both LL and calibre accessible to external network whereas right now only calibre is for its content server

philborman commented 7 years ago

ok, I will take a look at the copy/move section tomorrow. It may be calibre reads the opf, maybe not, depending on it's settings as I said before.

I think if you have two versions of calibre, one inside the lazylibrarian docker and one in it's own docker, only the standalone one will be exposed to outside. The one inside lazylibrarian docker will only be used by lazylibrarian (actually only the calibredb part) because the calibredb inside the lazylibrarian docker only exists inside that docker, in the same way the private lazylibrarian version of ghostscript only exists for lazylibrarian.

I have asked linuxserver.io for clarification, they are usually very helpful.

I had a quick look on calibres site and it seems you can install it with a single command... sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.py | sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"

CHBMB commented 7 years ago

@philborman is correct, if calibredb is visible to the docker container and appropriate permissions and dependencies are satisfied there's no reason it can't run.

CHBMB commented 7 years ago

Hang on, are we talking about a binary component of calibre?

If so: -v /lazylibrarian/calibredb:/usr/sbin/calibredb

ie mount the file in the same manner as a folder is mounted in docker.

CHBMB commented 7 years ago

You of course would need to get the calibredb binary from somewhere, as well as satisfy all the dependencies etc as I mentioned before.

ethanopp commented 7 years ago

@CHBMB I think that mount command you shared may solve all of this - will test as soon as I'm home!

ethanopp commented 7 years ago

@CHBMB It looks like there is a docker that already does this? https://hub.docker.com/r/thraxis/lazylibrarian-calibre/

Is this a one of the linuxserver.io stacks? The config parameters look identical...

I know aptalca calibre-rdp (calibre docker I'm running) isn't "technically" on the linuxserver.io repo but the dev works very closely with you guys...

CHBMB commented 7 years ago

No, nothing to do with us, guy just ripped off our logo and stuff. aptalca is one of our members but still has some solitary repos, as does sparklyballs.

If that works for you that's great, but I need to make it perfectly clear we won't support it, you have to contact that docker author.

ethanopp commented 7 years ago

Ahh that's a shame - Unfortunately I do not have the skillset to confidently customize the linuxserver.io docker to add the binary to it, otherwise I would. I'm not crazy about the fact that its a rip-off with no props given to the linuxserver team, but I think this may have to be the route I go given that it satisfies my need and I'm not sure of any other relatively straight-forward options... Thanks for all the awesome other containers though! :D

@philborman I did just test this out and confirm it solves my issues mentioned above - no duplicates anywhere... I did however notice a few other downsides...

Could you please confirm if the following behavior is expected of calibredb import program, or perhaps an issue with the thraxis docker? - I realize this may be more of a calibre question than LL at this point but appreciate any info you can share if you are knowledgeable of it:

  1. Although the calibre directories have been set up correctly, calibre does not see the added books until I restart/refresh it

  2. With the auto-add, calibre was able to auto convert to the output format (for me .mobi). I have this option selected in both the "Adding actions" and "Automatic Adding" settings in calibre but it appears to not be working even after the refresh

Thanks again in advance for all your help ...almost there!

philborman commented 7 years ago
  1. can you check the debug log when you use calibredb, you should see a line saying something like "notifying calibre of the change". Maybe this doesn't work in docker?
  2. There is no option in calibredb to do this, you either need to download mobi (set that as the only format in ebook types) or use a lazylibrarian custom_notifier. There is an example one supplied with lazylibrarian that converts epub to mobi and vice-versa
ethanopp commented 7 years ago

Okay I'll give the example_ebook_convert.py script a go... Here's my logs:

Timestamp Level Message
2017-07-22 15:43:33 DEBUG Restart processDir job in 10 minutes
2017-07-22 15:43:33 DEBUG Stop processDir job
2017-07-22 15:43:33 INFO 1 book/mag processed.
2017-07-22 15:43:33 DEBUG Snatched direct Amy Poehler - Yes Please LL.(20910157) is not in download directory
2017-07-22 15:43:33 DEBUG /downloads/Amy Poehler - Yes Please LL.(20910157) is not a directory?
2017-07-22 15:43:33 INFO Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads
2017-07-22 15:43:33 DEBUG Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT
2017-07-22 15:43:33 INFO Successfully processed: Yes Please - Amy Poehler
2017-07-22 15:43:33 DEBUG Deleted Amy Poehler - Yes Please LL.(20910157), direct from direct
2017-07-22 15:43:33 ERROR processExtras: No dest_file supplied
2017-07-22 15:43:33 DEBUG Processed direct: Yes Please - Amy Poehler, http://download.libgen.io/get.php?md5=4515217CF5F7C85B8DA12383618EC287&key=BWBNKBGAXWSKNCOJ
2017-07-22 15:43:33 INFO Library scan complete
2017-07-22 15:43:33 DEBUG Updating bookcounts for 1 author
2017-07-22 15:43:33 INFO 2 files processed
2017-07-22 15:43:33 INFO 1/1 new/modified eBooks found and added to the database
2017-07-22 15:43:33 DEBUG Amy Poehler Yes Please (9780062268372) matched Snatched BookID 20910157, [Amy Poehler][Yes Please]
2017-07-22 15:43:33 WARNING Updating book location for Amy Poehler Yes Please (9780062268372) from /books/Amy Poehler/Yes Please (118)/Yes Please - Amy Poehler.epub to /books/Amy Poehler/Yes Please (119)/Yes Please - Amy Poehler.epub
2017-07-22 15:43:33 DEBUG Link to preferred type epub: /books/Amy Poehler/Yes Please (119)/Yes Please - Amy Poehler.epub
2017-07-22 15:43:33 DEBUG Fuzz match partial [99] [Yes Please (9780062268372)] [Yes Please]
2017-07-22 15:43:33 DEBUG Searching 1 book by [Amy Poehler] in database for [Yes Please (9780062268372)]
2017-07-22 15:43:33 DEBUG Searching database for [Yes Please (9780062268372)] by [Amy Poehler]
2017-07-22 15:43:33 DEBUG Preferred authorname changed from [Poehler, Amy] to [Amy Poehler]
2017-07-22 15:43:33 DEBUG Formatted authorname [Poehler, Amy] to [Amy Poehler]
2017-07-22 15:43:33 DEBUG book meta [] [en] [Poehler, Amy] [Yes Please (9780062268372)] [epub]
2017-07-22 15:43:33 DEBUG [/books/Amy Poehler/] Now scanning subdirectory Yes Please (119)
2017-07-22 15:43:33 INFO Scanning eBook directory: /books/Amy Poehler/
2017-07-22 15:43:33 DEBUG /opt/calibre/calibredb identifier reports: No write acces to /root/.config/calibre using a temporary dir instead Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-22T19:43:27+00:00 Published : 0101-01-01T00:00:00+00:00 Identifiers : goodreads:20910157
2017-07-22 15:43:32 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'identifiers:goodreads:20910157', '--with-library', u'/books', '119']
2017-07-22 15:43:32 DEBUG /opt/calibre/calibredb title reports: No write acces to /root/.config/calibre using a temporary dir instead Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-22T19:43:27+00:00 Published : 0101-01-01T00:00:00+00:00
2017-07-22 15:43:31 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'title:Yes Please', '--with-library', u'/books', '119']
2017-07-22 15:43:31 DEBUG /opt/calibre/calibredb author reports: No write acces to /root/.config/calibre using a temporary dir instead Title : Yes Please (9780062268372) Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-22T19:43:27+00:00 Published : 0101-01-01T00:00:00+00:00
2017-07-22 15:43:29 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'authors:Amy Poehler', '--with-library', u'/books', '119']
2017-07-22 15:43:29 DEBUG Calibre ID: 119
2017-07-22 15:43:29 DEBUG /opt/calibre/calibredb reports: No write acces to /root/.config/calibre using a temporary dir instead Added book ids: 119
2017-07-22 15:43:26 DEBUG [u'/opt/calibre/calibredb', 'add', '-1', u'--with-library=/books', u'/downloads/Amy Poehler - Yes Please LL.(20910157)']
2017-07-22 15:43:26 DEBUG Importing Yes Please - Amy Poehler into calibre library
2017-07-22 15:43:26 DEBUG Processing eBook 20910157
2017-07-22 15:43:26 DEBUG Found match (100%): /downloads/Amy Poehler - Yes Please LL.(20910157) for eBook Amy Poehler - Yes Please LL.(20910157)
2017-07-22 15:43:26 DEBUG Found folder (100%) [/downloads/Amy Poehler - Yes Please LL.(20910157)] for eBook Amy Poehler - Yes Please
2017-07-22 15:43:26 INFO Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads
2017-07-22 15:43:26 DEBUG Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT
2017-07-22 15:43:26 DEBUG Found 2 files marked "Snatched"
2017-07-22 15:43:26 DEBUG Checking 1 file in /downloads
2017-07-22 15:43:23 INFO Log cleared, level set to [2]- Log Directory is [/config/log]
philborman commented 7 years ago

No write acces to /root/.config/calibre Looks like you need to make it rw ?

ethanopp commented 7 years ago

I saw those in the DEBUG but was not sure if that was directly causing the "processExtras: No dest_file supplied" error...

I believe the /root/.config/calibre directory is on the container side, and isn't mounted through the /config volume, so every time the image is pulled to refresh this would have to manually be updated - I opened an issue on the docker specific page as this permission should probably be set by default: https://github.com/Thraxis/docker-lazylibrarian-calibre/issues/2

On a side note - I tried SSHing into the docker and couldn't find this dir manually...

In terms of the notifier - I am assuming it is just not working because of this issue above, however I've downloaded the script, updated the ebook-converter full path, set the permissions to rwxr-xr-- and updated the LL settings...

Hoping to get a quick response there as it should be a quick fix for the dev to just apply the write permissions.

Thraxis commented 7 years ago

It should be fixed now

philborman commented 7 years ago

A few more thoughts. The "processExtras: No dest_file supplied" error... is because we think calibre failed to import the book properly, and the notifier isn't running for the same reason, no new book to notify you about.

It looks like calibredb talks to a running calibre instance through /root/.config/calibre so I think you probably need to run the thraxis container for both calibre and lazylibrarian and stop the standalone calibre one. That could also be why you are having to restart/refresh calibre after adding books, the "wrong" calibre is being told about changes.

ethanopp commented 7 years ago

@philborman - Tested this again with calibre shutdown, still getting the "processExtras: No dest_file supplied" error

philborman commented 7 years ago

I've just reworked the calibredb section, more progress messages on debug, and a different way of getting dest_file (work it out ourselves instead of asking calibre). Might solve the problem?

ethanopp commented 7 years ago

Still on the Thraxis docker (re-pulled image) but not sure if your changes made it through yet...

I did do a full library scan and author refresh and don't seem to be getting the "processExtras: No dest_file supplied" error anymore...

...Now this though:

Timestamp Level Message
2017-07-23 12:44:41 ERROR Unhandled exception in processDir: Traceback (most recent call last): File "/app/lazylibrarian/lazylibrarian/postprocess.py", line 459, in processDir global_name, book['BookID'], book_type) File "/app/lazylibrarian/lazylibrarian/postprocess.py", line 1044, in processDestination imported = LibraryScan(calibre_dir, remove=remove) # rescan whole authors directory UnboundLocalError: local variable 'remove' referenced before assignment
2017-07-23 12:44:41 DEBUG Calibre trying directory [/books/Amy Poehler/Yes Please - Amy Poehler (210)]
2017-07-23 12:44:41 DEBUG /opt/calibre/calibredb identifier reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T16:44:35+00:00 Published : 0101-01-01T00:00:00+00:00 Identifiers : goodreads:20910157
2017-07-23 12:44:40 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'identifiers:goodreads:20910157', '--with-library', u'/books', '210']
2017-07-23 12:44:40 DEBUG /opt/calibre/calibredb title reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T16:44:35+00:00 Published : 0101-01-01T00:00:00+00:00
2017-07-23 12:44:39 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'title:Yes Please', '--with-library', u'/books', '210']
2017-07-23 12:44:39 DEBUG /opt/calibre/calibredb author reports: Title : Yes Please (9780062268372) Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T16:44:35+00:00 Published : 0101-01-01T00:00:00+00:00
2017-07-23 12:44:37 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'authors:Amy Poehler', '--with-library', u'/books', '210']
2017-07-23 12:44:37 DEBUG Calibre ID: 210
2017-07-23 12:44:37 DEBUG /opt/calibre/calibredb reports: Added book ids: 210
2017-07-23 12:44:35 DEBUG [u'/opt/calibre/calibredb', 'add', '-1', u'--with-library=/books', u'/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)']
2017-07-23 12:44:35 DEBUG Importing Yes Please - Amy Poehler into calibre library
2017-07-23 12:44:35 DEBUG Processing eBook 20910157
2017-07-23 12:44:35 DEBUG Found match (100%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) for eBook Amy Poehler - Yes Please LL.(20910157)
2017-07-23 12:44:35 DEBUG Found folder (100%) [/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)] for eBook Amy Poehler - Yes Please
2017-07-23 12:44:35 INFO Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads/completed/LazyLibrarian
2017-07-23 12:44:35 DEBUG Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT
2017-07-23 12:44:35 DEBUG Closest match (26%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 12:44:35 DEBUG Snatched direct David Baldacci - Memory Man LL.(23153154) is not in download directory
2017-07-23 12:44:35 INFO Looking for eBook David Baldacci - Memory Man LL.(23153154) in /downloads/completed/LazyLibrarian
2017-07-23 12:44:35 DEBUG David Baldacci - Memory Man LL.(23153154) was sent to DIRECT
2017-07-23 12:44:35 DEBUG Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 12:44:35 DEBUG Snatched torznab Tarryn Fisher - Marrow LL.(25379213) is not in download directory
2017-07-23 12:44:35 INFO Looking for eBook Tarryn Fisher - Marrow LL.(25379213) in /downloads/completed/LazyLibrarian
2017-07-23 12:44:35 DEBUG getTorrentFolder: No response from transmission
2017-07-23 12:44:35 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Tarryn Fisher - Marrow LL.(25379213) was sent to TRANSMISSION
2017-07-23 12:44:35 DEBUG Closest match (18%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 12:44:35 DEBUG Snatched torznab Megan Miranda - All the Missing Girls LL.(27276324) is not in download directory
2017-07-23 12:44:35 INFO Looking for eBook Megan Miranda - All the Missing Girls LL.(27276324) in /downloads/completed/LazyLibrarian
2017-07-23 12:44:35 DEBUG getTorrentFolder: No response from transmission
2017-07-23 12:44:35 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Megan Miranda - All the Missing Girls LL.(27276324) was sent to TRANSMISSION
2017-07-23 12:44:35 DEBUG Closest match (42%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 12:44:35 DEBUG Snatched torznab Paulo Coelho - The Alchemist LL.(865) is not in download directory
2017-07-23 12:44:35 INFO Looking for eBook Paulo Coelho - The Alchemist LL.(865) in /downloads/completed/LazyLibrarian
2017-07-23 12:44:35 DEBUG getTorrentFolder: No response from transmission
2017-07-23 12:44:35 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Paulo Coelho - The Alchemist LL.(865) was sent to TRANSMISSION
2017-07-23 12:44:35 DEBUG Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 12:44:35 DEBUG Snatched magnet Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK is not in download directory
2017-07-23 12:44:35 INFO Looking for eBook Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK in /downloads/completed/LazyLibrarian
2017-07-23 12:44:35 DEBUG getTorrentFolder: No response from transmission
2017-07-23 12:44:35 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 12:44:35 DEBUG Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK was sent to TRANSMISSION
2017-07-23 12:44:35 DEBUG Found 6 files marked "Snatched"
2017-07-23 12:44:35 DEBUG Checking 1 file in /downloads/completed/LazyLibrarian
2017-07-23 12:44:33 INFO Log cleared, level set to [2]- Log Directory is [/config/log]
philborman commented 7 years ago

ah- just fixed that. New version out shortly

ethanopp commented 7 years ago

okay great - I also noticed the new functionality you added for "Autoadd eBook only", thanks for getting this added!

1 thought around it - would it be possible to have this also apply when not using either of the "Calibre Auto Add Directory" or "Calibredb import program" functionalities?

By setting the "eBook Foldername Pattern" the same as the "Book Library Folder", LL will currently move all of the downloaded files directly to the main /book directory, which Calibre monitors only for the book files to work its auto-add magic...

By doing this, we can remove LL from any of the calibre processing - and on the next library scan, LL will see the books in the dirs that calibre automatically sets up.

If we can set LL to only send the .epub over then there wont be any duplicates .opf/.jpgs leftover in the /book dir. Similarly, there wont be any duplicate epubs left over in the download dir as they are being moved to /book and not copied.

ethanopp commented 7 years ago

Just tested out calibredb on the new version.

Everything seems to be getting placed in the correct places, only problem is calibre still doesn't show the book until the library is manually refreshed or calibre is restarted.

2017-07-23 19:10:52 DEBUG Restart processDir job in 10 minutes
2017-07-23 19:10:52 DEBUG Stop processDir job
2017-07-23 19:10:52 INFO 1 book/mag processed.
2017-07-23 19:10:52 DEBUG Snatched direct Amy Poehler - Yes Please LL.(20910157) is not in download directory
2017-07-23 19:10:52 DEBUG /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) is not a directory?
2017-07-23 19:10:52 INFO Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads/completed/LazyLibrarian
2017-07-23 19:10:52 DEBUG Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT
2017-07-23 19:10:51 DEBUG Pushbullet devices: None
2017-07-23 19:10:51 DEBUG Pushbullet api: XXXXXXXXXXXXXXXXXXXX
2017-07-23 19:10:51 DEBUG Pushbullet message: eBook Yes Please - Amy Poehler from libgen.io/search.php at 2017-07-23 19:10:51
2017-07-23 19:10:51 DEBUG Pushbullet event: Added to Library
2017-07-23 19:10:51 DEBUG Pushbullet: Sending notification eBook Yes Please - Amy Poehler from libgen.io/search.php at 2017-07-23 19:10:51
2017-07-23 19:10:51 INFO Successfully processed: Yes Please - Amy Poehler
2017-07-23 19:10:51 DEBUG Deleted Amy Poehler - Yes Please LL.(20910157), direct from direct
2017-07-23 19:10:51 DEBUG Calibre should have created the extras for us
2017-07-23 19:10:51 DEBUG Processed direct: Yes Please - Amy Poehler, http://download.libgen.io/get.php?md5=4515217CF5F7C85B8DA12383618EC287&key=JBXFJ1JUJOBJO4Y3
2017-07-23 19:10:51 INFO Library scan complete
2017-07-23 19:10:51 DEBUG Updating bookcounts for 1 author
2017-07-23 19:10:51 INFO 2 files processed
2017-07-23 19:10:51 INFO 1/1 new/modified eBooks found and added to the database
2017-07-23 19:10:51 DEBUG Amy Poehler Yes Please (9780062268372) matched Snatched BookID 20910157, [Amy Poehler][Yes Please]
2017-07-23 19:10:51 WARNING Updating book location for Amy Poehler Yes Please (9780062268372) from /books/Amy Poehler/Yes Please (186)/Yes Please - Amy Poehler.epub to /books/Amy Poehler/Yes Please (226)/Yes Please - Amy Poehler.epub
2017-07-23 19:10:51 DEBUG Librarysync link to preferred type epub: /books/Amy Poehler/Yes Please (226)/Yes Please - Amy Poehler.epub
2017-07-23 19:10:50 DEBUG Fuzz match partial [99] [Yes Please (9780062268372)] [Yes Please]
2017-07-23 19:10:50 DEBUG Searching 1 book by [Amy Poehler] in database for [Yes Please (9780062268372)]
2017-07-23 19:10:50 DEBUG Searching database for [Yes Please (9780062268372)] by [Amy Poehler]
2017-07-23 19:10:50 DEBUG Preferred authorname changed from [Poehler, Amy] to [Amy Poehler]
2017-07-23 19:10:50 DEBUG Formatted authorname [Poehler, Amy] to [Amy Poehler]
2017-07-23 19:10:50 DEBUG book meta [] [en] [Poehler, Amy] [Yes Please (9780062268372)] [epub]
2017-07-23 19:10:50 DEBUG [/books/Amy Poehler/] Now scanning subdirectory Yes Please (226)
2017-07-23 19:10:50 INFO Scanning eBook directory: /books/Amy Poehler/
2017-07-23 19:10:50 DEBUG Calibre trying directory [/books/Amy Poehler/Yes Please - Amy Poehler (226)]
2017-07-23 19:10:50 DEBUG /opt/calibre/calibredb identifier reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T23:10:44+00:00 Published : 0101-01-01T00:00:00+00:00 Identifiers : goodreads:20910157
2017-07-23 19:10:49 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'identifiers:goodreads:20910157', '--with-library', u'/books', '226']
2017-07-23 19:10:49 DEBUG /opt/calibre/calibredb title reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T23:10:44+00:00 Published : 0101-01-01T00:00:00+00:00
2017-07-23 19:10:48 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'title:Yes Please', '--with-library', u'/books', '226']
2017-07-23 19:10:48 DEBUG /opt/calibre/calibredb author reports: Title : Yes Please (9780062268372) Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T23:10:44+00:00 Published : 0101-01-01T00:00:00+00:00
2017-07-23 19:10:46 DEBUG [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'authors:Amy Poehler', '--with-library', u'/books', '226']
2017-07-23 19:10:46 DEBUG Calibre ID: 226
2017-07-23 19:10:46 DEBUG /opt/calibre/calibredb reports: Added book ids: 226
2017-07-23 19:10:43 DEBUG [u'/opt/calibre/calibredb', 'add', '-1', u'--with-library=/books', u'/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)']
2017-07-23 19:10:43 DEBUG Importing Yes Please - Amy Poehler into calibre library
2017-07-23 19:10:43 DEBUG Processing eBook 20910157
2017-07-23 19:10:43 DEBUG Found match (100%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) for eBook Amy Poehler - Yes Please LL.(20910157)
2017-07-23 19:10:43 DEBUG Found folder (100%) [/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)] for eBook Amy Poehler - Yes Please
2017-07-23 19:10:43 INFO Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads/completed/LazyLibrarian
2017-07-23 19:10:43 DEBUG Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT
2017-07-23 19:10:43 DEBUG Closest match (26%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 19:10:43 DEBUG Snatched direct David Baldacci - Memory Man LL.(23153154) is not in download directory
2017-07-23 19:10:43 INFO Looking for eBook David Baldacci - Memory Man LL.(23153154) in /downloads/completed/LazyLibrarian
2017-07-23 19:10:43 DEBUG David Baldacci - Memory Man LL.(23153154) was sent to DIRECT
2017-07-23 19:10:43 DEBUG Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 19:10:43 DEBUG Snatched torznab Tarryn Fisher - Marrow LL.(25379213) is not in download directory
2017-07-23 19:10:43 INFO Looking for eBook Tarryn Fisher - Marrow LL.(25379213) in /downloads/completed/LazyLibrarian
2017-07-23 19:10:43 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:43 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:38 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:38 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Tarryn Fisher - Marrow LL.(25379213) was sent to TRANSMISSION
2017-07-23 19:10:33 DEBUG Closest match (18%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 19:10:33 DEBUG Snatched torznab Megan Miranda - All the Missing Girls LL.(27276324) is not in download directory
2017-07-23 19:10:33 INFO Looking for eBook Megan Miranda - All the Missing Girls LL.(27276324) in /downloads/completed/LazyLibrarian
2017-07-23 19:10:33 DEBUG getTorrentFolder: No response from transmission
2017-07-23 19:10:33 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Megan Miranda - All the Missing Girls LL.(27276324) was sent to TRANSMISSION
2017-07-23 19:10:33 DEBUG Closest match (42%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 19:10:33 DEBUG Snatched torznab Paulo Coelho - The Alchemist LL.(865) is not in download directory
2017-07-23 19:10:33 INFO Looking for eBook Paulo Coelho - The Alchemist LL.(865) in /downloads/completed/LazyLibrarian
2017-07-23 19:10:33 DEBUG getTorrentFolder: No response from transmission
2017-07-23 19:10:33 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Paulo Coelho - The Alchemist LL.(865) was sent to TRANSMISSION
2017-07-23 19:10:33 DEBUG Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)
2017-07-23 19:10:33 DEBUG Snatched magnet Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK is not in download directory
2017-07-23 19:10:33 INFO Looking for eBook Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK in /downloads/completed/LazyLibrarian
2017-07-23 19:10:33 DEBUG getTorrentFolder: No response from transmission
2017-07-23 19:10:33 DEBUG Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc
2017-07-23 19:10:33 DEBUG Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK was sent to TRANSMISSION
2017-07-23 19:10:33 DEBUG Found 7 files marked "Snatched"
2017-07-23 19:10:33 DEBUG Checking 1 file in /downloads/completed/LazyLibrarian
2017-07-23 19:10:29 INFO Log cleared, level set to [2]- Log Directory is [/config/log]
Thraxis commented 7 years ago

Yes that would be the correct behavior currently.

ethanopp commented 7 years ago

Ah okay thanks for clarifying... Is that just for Calibre?

For example if I were running COPS or Calibre-Web, would those have to be restarted as well for the books to show?

Thraxis commented 7 years ago

I just imported a book and COPS did after a little bit display it in the UI. but that could have been due to my changing from the default template to the bootstrap one

ethanopp commented 7 years ago

Any way to pass a "refresh" command through calibredb after a book is added to be sure?

philborman commented 7 years ago

There doesn't seem to be a separate command to do this, it should happen automatically with each calibredb command, see log below. After each command you get a message from calibredb "Notifying calibre of the change". I guess if you are not seeing this message, calibredb doesn't know calibre is running, or can't talk to it. I don't know how it checks, but I have calibre running on the same machine as calibredb, so they must communicate somehow, and in a way that doesn't need any configuring since I have never done any. Maybe a socket? Maybe it looks for the pid of calibre and sends a signal?

There is another way of talking to a calibre content server with calibredb, using http://host:port/#library_id which in theory lets calibredb talk to calibre on another machine (or in another docker) but I haven't managed to get that to work. The documentation says http://host:port/#- should give a list of library ids but I just get "Not Found"

24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : Found match (100%): /media/Popcorn/Download/complete/Books/Prince Lestat and the Realms of Atlantis LL.(23267873) for eBook Prince Lestat and the Realms of Atlantis 24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : Processing eBook 23267873 24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : Importing Prince Lestat and the Realms of Atlantis - Anne Rice into calibre library 24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : [u'calibredb', 'add', '-1', u'--with-library=/media/Popcorn/Calibre Library/', u'/media/Popcorn/Download/complete/Books/Prince Lestat and the Realms of Atlantis LL.(23267873)'] 24-Jul-2017 12:47:32 - DEBUG :: POSTPROCESS : calibredb reports: Backing up metadata Added book ids: 1851 Notifying calibre of the change

24-Jul-2017 12:47:32 - DEBUG :: POSTPROCESS : Calibre ID: 1851 24-Jul-2017 12:47:32 - DEBUG :: POSTPROCESS : [u'calibredb', 'set_metadata', '--field', u'authors:Anne Rice', '--with-library', u'/media/Popcorn/Calibre Library/', '1851'] 24-Jul-2017 12:47:44 - DEBUG :: POSTPROCESS : calibredb author reports: Title : Prince Lestat and the Realms of Atlantis Title sort : Prince Lestat and the Realms of Atlantis Author(s) : Anne Rice [Rice, Anne] Publisher : Knopf Doubleday Publishing Group Languages : eng Timestamp : 2017-07-24T10:47:22+00:00 Published : 2016-11-28T23:00:00+00:00 Comments : From Anne Rice, conjurer of the beloved best sellers Interview with a Vampire and Prince Lestat, an ambitious and exhilarating new novel of utopian vision and power, uniting the ancient worlds and the legends of the Vampire Chronicles.

"In my dreams, I saw a city fall into the sea. I heard the cries of thousands. I saw flames that outshone the lamps of heaven. And all the world was shaken . . ." —Anne Rice, Prince Lestat and the Realms of Atlantis

At the novel's center: the vampire Lestat de Lioncourt, hero, leader, inspirer, irresistible force, irrepressible spirit, battling (and ultimately reconciling with) a strange otherworldly form that has somehow taken possession of Lestat's undead body and soul. This ancient and mysterious power and unearthly spirit of vampire lore has all the force, history, and insidious reach of the unknowable Universe.
It is through this spirit, previously considered benign for thousands of vampire... Backing up metadata Notifying calibre of the change

Thraxis commented 7 years ago

The instructions are a little unclear on how the remote content server works with calibredb.

First of all you must have the Calibre Content Server running. Just having the GUI running does not start the Content Server. Click on Connect/Share and Start the Content Server. Note the IP address and port.

The command-line for calibredb would then be:

/opt/calibre/calibredb --with-library="http://:/#-" list

Which will give you the list of library names, something like as follows: FanFiction FanFiction_(Complete) SciFi

/opt/calibre/calibredb --with-library="http://:/#SciFi" list Will then list the books in the SciFi library (which is my default for adding to Calibre)

I'm not sure how usefull this would actually be as there is no "refresh" option and the remote commands only seem to allow you to control a remote Content Server, which means that both the system LL is on and the Calibre Content Server are on need to have access to the path where the book to be added is located, there is no provision for "uploading" the file to the Content Server.

So you pretty much either have to run dockers with identical paths, or you have to have the path mapping for the remote system.

philborman commented 7 years ago

I have a running calibre-server on port 9000 and can talk to it using a web browser, but the calibredb commands just don't work for me. calibredb --with-library="http://localhost:9000/#-" list gives the Usage page

calibredb list --with-library="http://localhost:9000/#-" id title authors (as if it's trying to list the library contents but can't find any, not listing the library ids)

calibredb list --with-library="http://localhost:9000/#Calibre Library" id title authors (same)

calibredb list --with-library="/home/phil/Calibre Library" lists all the books

ethanopp commented 7 years ago

@philborman - I pulled the update for the "Autoadd eBook only" but it still looks like when not using the claibre autoadd or calibredb the extra files are still being moved to the main book dir.

philborman commented 7 years ago

I did try that way, but there are problems because the "main book dir" is your lazylibrarian library. If you don't put a cover image in the library, lazylibrarian will download a new one, as it needs the cover for it's own display purposes. We could miss out the opf file, as at the moment lazylibrarian doesn't use it.

ethanopp commented 7 years ago

Ah I see - what is interesting here though is that when calibre does its own "Auto Add" by seeing the .epub in /books, that .jpg is somehow making it into the new dir it sets up.

I am not sure if calibre is using the .jpg that LL downloaded and copying it while moving the .epub to the new dir, or if it is downloading the .jpg on its own once it moves the .epub... Either way the .jpg is making its way into the dir that claibre sets up, and LL can read that .jpg because they still show in LL after I delete those .jpgs from the main /book dir and do a library scan.

philborman commented 7 years ago

Could be either calibre copying it or downloading it's own, hard to say, depends on calibre settings, among other things. Easy way to find out is to deliberately put the wrong image in there and see if calibre picks it up. Calibre will also read our opf file, again depending on calibre settings, and also depending on embedded book data - seems calibre prefers embedded data to opf data, but will read other fields from the opf if they are not present in embedded data. Also our opf includes a link to the book image, so calibre could pick it up from there.