Closed ethanopp closed 7 years ago
I did a quick google for "text file busy" and it seems to crop up for all sorts of reasons. Doesn't necessarily mean a permissions issue, you can sometimes get that error message if unable to delete/rename a file, some other program using a file, etc. Unfortunately the small debug log snipped doesn't give enough info to go on. Can you post larger segment showing the whole postprocessor run on that book (or use the new log creator on lazylibrarian log page). Does it only happen on the one book? What version of calibredb in case that's relevant. I use calibredb myself when importing and haven't seen this message, so don't really know where to start looking. I was on calibre 2.75 until a few days ago, just switched to calibre 3.4 but haven't tested that much yet.
Just updated calibre to 3.4 and LL to 242dea93c74f16c2c3149d22cae6a817fa962cf5
Same issue... Sorry for not attaching the firs time, I was going to post but of course as soon as tried to replicate manually none of the books would download... Waited for one to finally get grabbed so I could grab just the post-processor log (attached now)
The first attempt was on a book that LL downloaded multiple file types of so it went into a directory, this second attempt only downloaded the single .epub so there was no directory created for it, but same issue both times with different books...
UPDATE: Doesnt seem like the save debug log worked as expected... Pasted here as well:
2017-07-20 09:25:51 | INFO | Debug log saved as /config/log/debug.zip |
---|---|---|
2017-07-20 09:25:51 | DEBUG | 20 lines written to /config/log/debug.log |
2017-07-20 09:25:51 | DEBUG | Redacted 0 passwords/apikeys |
2017-07-20 09:22:31 | INFO | Debug log saved as /config/log/debug.zip |
2017-07-20 09:22:31 | DEBUG | 20 lines written to /config/log/debug.log |
2017-07-20 09:22:31 | DEBUG | Redacted 0 passwords/apikeys |
2017-07-20 09:22:11 | DEBUG | Restart processDir job in 10 minutes |
2017-07-20 09:22:11 | DEBUG | Stop processDir job |
2017-07-20 09:22:11 | INFO | 0 books/mags processed. |
2017-07-20 09:22:11 | DEBUG | Closest match (30%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:22:11 | DEBUG | Snatched torznab Walter Isaacson - Steve Jobs LL.(11084145) is not in download directory |
2017-07-20 09:22:11 | INFO | Looking for eBook Walter Isaacson - Steve Jobs LL.(11084145) in /downloads/completed/LazyLibrarian |
2017-07-20 09:22:11 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:11 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:06 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:06 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:01 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:01 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:01 | DEBUG | Walter Isaacson - Steve Jobs LL.(11084145) was sent to TRANSMISSION |
2017-07-20 09:22:01 | DEBUG | Closest match (30%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:22:01 | DEBUG | Snatched torznab Gone Girl - Gillian Flynn is not in download directory |
2017-07-20 09:22:01 | INFO | Looking for eBook Gone Girl - Gillian Flynn in /downloads/completed/LazyLibrarian |
2017-07-20 09:22:00 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:00 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:00 | DEBUG | Gone Girl - Gillian Flynn was sent to TRANSMISSION |
2017-07-20 09:22:00 | DEBUG | Closest match (33%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:22:00 | DEBUG | Snatched torznab Robert Galbraith - The Cuckoos Calling LL.(16160834) is not in download directory |
2017-07-20 09:22:00 | INFO | Looking for eBook Robert Galbraith - The Cuckoos Calling LL.(16160834) in /downloads/completed/LazyLibrarian |
2017-07-20 09:22:00 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:22:00 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:55 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:55 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:50 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:50 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:50 | DEBUG | Robert Galbraith - The Cuckoos Calling LL.(16160834) was sent to TRANSMISSION |
2017-07-20 09:21:50 | DEBUG | Closest match (35%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:50 | DEBUG | Snatched torznab The Chaperone by Laura Moriarty is not in download directory |
2017-07-20 09:21:50 | INFO | Looking for eBook The Chaperone by Laura Moriarty in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:50 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:50 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:50 | DEBUG | The Chaperone by Laura Moriarty was sent to TRANSMISSION |
2017-07-20 09:21:50 | DEBUG | Closest match (41%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:50 | DEBUG | Snatched torznab Amy Tan - The Valley of Amazement LL.(17383934) is not in download directory |
2017-07-20 09:21:50 | INFO | Looking for eBook Amy Tan - The Valley of Amazement LL.(17383934) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:50 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:50 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:45 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:45 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:40 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:40 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:40 | DEBUG | Amy Tan - The Valley of Amazement LL.(17383934) was sent to TRANSMISSION |
2017-07-20 09:21:40 | DEBUG | Closest match (28%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:40 | DEBUG | Snatched torznab Elizabeth Smart - My Story LL.(17287028) is not in download directory |
2017-07-20 09:21:40 | INFO | Looking for eBook Elizabeth Smart - My Story LL.(17287028) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:40 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-20 09:21:40 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:40 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:40 | DEBUG | Elizabeth Smart - My Story LL.(17287028) was sent to TRANSMISSION |
2017-07-20 09:21:40 | DEBUG | Closest match (35%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:40 | DEBUG | Snatched torrent Jeannette Walls - The Glass Castle LL.(7445) is not in download directory |
2017-07-20 09:21:40 | INFO | Looking for eBook Jeannette Walls - The Glass Castle LL.(7445) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:40 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:40 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:35 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:35 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:30 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:30 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:30 | DEBUG | Jeannette Walls - The Glass Castle LL.(7445) was sent to TRANSMISSION |
2017-07-20 09:21:30 | DEBUG | Closest match (23%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:30 | DEBUG | Snatched torznab Kristin Hannah - Between Sisters LL.(217432) is not in download directory |
2017-07-20 09:21:30 | INFO | Looking for eBook Kristin Hannah - Between Sisters LL.(217432) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:30 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:30 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:25 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:25 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:20 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:20 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:20 | DEBUG | Kristin Hannah - Between Sisters LL.(217432) was sent to TRANSMISSION |
2017-07-20 09:21:20 | DEBUG | Closest match (35%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:20 | DEBUG | Snatched torznab Revolution - Jennifer Donnelly.epub is not in download directory |
2017-07-20 09:21:20 | INFO | Looking for eBook Revolution - Jennifer Donnelly.epub in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:20 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:20 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:20 | DEBUG | Revolution - Jennifer Donnelly.epub was sent to TRANSMISSION |
2017-07-20 09:21:20 | DEBUG | Closest match (17%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:20 | DEBUG | Snatched torznab Tracey Martin - Another Little Piece of My Heart LL.(18336825) is not in download directory |
2017-07-20 09:21:20 | INFO | Looking for eBook Tracey Martin - Another Little Piece of My Heart LL.(18336825) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:20 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:20 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:15 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:15 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:10 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:10 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:10 | DEBUG | Tracey Martin - Another Little Piece of My Heart LL.(18336825) was sent to TRANSMISSION |
2017-07-20 09:21:10 | WARNING | Residual files remain in /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.fail |
2017-07-20 09:21:10 | ERROR | Postprocessing for The Girl with the Lower Back Tattoo - Amy Schumer has failed: calibredb failed, Text file busy |
2017-07-20 09:21:10 | DEBUG | [u'/books/metadata.db', 'add', '-1', u'--with-library=/books', u'/downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer'] |
2017-07-20 09:21:10 | DEBUG | Importing The Girl with the Lower Back Tattoo - Amy Schumer into calibre library |
2017-07-20 09:21:10 | DEBUG | One format import, best match = epub |
2017-07-20 09:21:10 | DEBUG | Processing eBook 29405093 |
2017-07-20 09:21:10 | DEBUG | Found match (100%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer for eBook Amy Schumer - The Girl with the Lower Back Tattoo LL.(29405093) |
2017-07-20 09:21:10 | DEBUG | Found folder (100%) [/downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer] for eBook The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:10 | INFO | Looking for eBook The Girl with the Lower Back Tattoo - Amy Schumer.epub in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:10 | DEBUG | TRANSMISSION Changing [Amy Schumer - The Girl with the Lower Back Tattoo LL.(29405093)] to [The Girl with the Lower Back Tattoo - Amy Schumer.epub] |
2017-07-20 09:21:10 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:10 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:10 | DEBUG | Amy Schumer - The Girl with the Lower Back Tattoo LL.(29405093) was sent to TRANSMISSION |
2017-07-20 09:21:10 | DEBUG | Closest match (26%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:10 | DEBUG | Snatched torznab Tarryn Fisher - Marrow LL.(25379213) is not in download directory |
2017-07-20 09:21:10 | INFO | Looking for eBook Tarryn Fisher - Marrow LL.(25379213) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:10 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:10 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:05 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:05 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:00 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:00 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:00 | DEBUG | Tarryn Fisher - Marrow LL.(25379213) was sent to TRANSMISSION |
2017-07-20 09:21:00 | DEBUG | Closest match (37%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:21:00 | DEBUG | Snatched torznab Louise Penny - A Trick of the Light LL.(10680329) is not in download directory |
2017-07-20 09:21:00 | INFO | Looking for eBook Louise Penny - A Trick of the Light LL.(10680329) in /downloads/completed/LazyLibrarian |
2017-07-20 09:21:00 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:21:00 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:55 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:55 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:50 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:50 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:50 | DEBUG | Louise Penny - A Trick of the Light LL.(10680329) was sent to TRANSMISSION |
2017-07-20 09:20:50 | DEBUG | Closest match (38%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:20:50 | DEBUG | Snatched torznab Dale Carnegie - How to Win Friends and Influence People LL.(4865) is not in download directory |
2017-07-20 09:20:50 | INFO | Looking for eBook Dale Carnegie - How to Win Friends and Influence People LL.(4865) in /downloads/completed/LazyLibrarian |
2017-07-20 09:20:50 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:50 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:45 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:45 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | Dale Carnegie - How to Win Friends and Influence People LL.(4865) was sent to TRANSMISSION |
2017-07-20 09:20:40 | DEBUG | Closest match (12%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:20:40 | DEBUG | Snatched torrent TheOmnivoresDilemma is not in download directory |
2017-07-20 09:20:40 | INFO | Looking for eBook TheOmnivoresDilemma in /downloads/completed/LazyLibrarian |
2017-07-20 09:20:40 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-20 09:20:40 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | TheOmnivoresDilemma was sent to TRANSMISSION |
2017-07-20 09:20:40 | DEBUG | Closest match (40%): /downloads/completed/LazyLibrarian/The Girl with the Lower Back Tattoo - Amy Schumer.epub |
2017-07-20 09:20:40 | DEBUG | Snatched torznab Jane_Harper-The_Dry-ABOOK-WEB-2017-PROLOG is not in download directory |
2017-07-20 09:20:40 | INFO | Looking for eBook Jane_Harper-The_Dry-ABOOK-WEB-2017-PROLOG in /downloads/completed/LazyLibrarian |
2017-07-20 09:20:40 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-20 09:20:40 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-20 09:20:40 | DEBUG | Jane_Harper-The_Dry-ABOOK-WEB-2017-PROLOG was sent to TRANSMISSION |
2017-07-20 09:20:40 | DEBUG | Found 16 files marked "Snatched" |
2017-07-20 09:20:40 | DEBUG | Checking 1 file in /downloads/completed/LazyLibrarian |
2017-07-20 09:20:37 | INFO | Log cleared, level set to [2]- Log Directory is [/config/log] |
Ah - that's it. The entry in lazylibrarian config is wrong. config->processing->Calibredb import program: should be the program, not the database. Calibre has a command line program shipped with it called "calibredb" which is what we need to pass to lazylibrarian. It might be that the program is in your system path, in which case just put calibredb in the box, or you may need to put the full path, eg /usr/bin/calibredb. Depends on how docker is set up. If you have access to your docker via ssh the command "which calibredb" will tell you where it is.
For reference, I am using the following docker for calibre: https://github.com/aptalca/docker-rdp-calibre
'which calibre' gives me the following: /usr/bin/calibre... went into the dir just to verify and found /usr/bin/calibredb
Updated the config in LL, restarted both calibre and LL, still getting same issue...
Then realized it is because both are in their own containers, so LL cannot see the the /usr/bin of calibre...
Whats the best way to allow LL docker to see the calibredb inside the calibre docker's /usr/bin?
UPDATE: just typing 'calibredb' seems to run the program fine in the calibre docker - still not sure how to get that over to LL though...
Also quick side question while I am going through all of this trouble - does the calibredb import program import the metadata (.opf, covers, etc.) that LL downloads? ...Kind of why I am going through all of this :D
Sorry, I don't use docker. There is another lazylibrarian docker at Linuxserver.io link at the bottom of lazylibrarian git page. Maybe that includes calibredb? Have you tried putting calibredb into your /books folder as lazylibrarian can see that?
That's actually the docker I am using - it seems that this may be more of a limitation of calibredb though so I may be going through all of this for nothing.
LL downloads .opf metadata with the books, and I was looking for a way to automatically import that into calibre... I was asking on the calibre IRC though and it seems they are saying that calibredb import program only imports the book files?
Am I getting some incorrect information here? Don't really see the value in LL downloading the metadata if there is no way to import it into calibre (at least for my setup) because calibre is going to have to redownload it all itself from within calibre.
Some book downloads already include the opf file and a cover, some don't. Lazylibrarian will create an opf file for those books that don't have one, but does not modify embedded book metadata or existing opf files. Whether calibre uses it or not depends on calibre settings, there is a lot of conflicting advice out there, but there is a calibre setting called something like "read metadata from file contents" . If this is on, calibre will read the book metadata for as much info as it can get, and then read any additional metadata from the opf file. It seems embedded book metadata takes preference over opf file, so if calibre got all it wanted from the book it won't read the opf.
If the "read metadata from files" is not set, calibre tries to work out the author and title from the filename and ignores the book metadata and opf file.
In lazylibrarian we use calibredb to send the book, cover, opf and then also force calibre to set the suthor and book title to what we expect, as calibre will use embedded data which might not match what we want.
With magazines, which are usually pdf files, there is no useful metadata and calibre happily uses our opf files.
Alright.. Thank you for all of this info - it has been extremely helpful!
I think for the most part I have been able to get to the end result I need with the use of calibre's auto add and LL's ability to copy everything into that dir.
I have 1 last small issue...
When LL finds a download with only 1 book it downloads only that file to the download dir - and does not create a sub directory: /downloads/bookname.epub
Once the post processing runs, the sub-dir gets set up correctly, downloads the metadata and copies the book there, but it keeps the original book file in the main download dir for seeding: /downloads/bookname.epub /downloads/book name/bookname.epub /downloads/book name/bookname.opf /downloads/book name/bookname.jpg
Once seeding is complete, rather then deleting the entire sub directory with all of the files, it only deletes the single book file from the main download dir, and I am left with duplicate files from the sub dir, as the auto add feature has copied this entire sub dir already to my calibre dir.
Is there a way to have LL do a "move" instead of "copy" for the auto add feature?
NOTE: I have "Keep original files" unchecked and "Import one format" checked
Just added some code to allow move rather than copy to autoadd if "keep original files" is unchecked. Not sure what this will do to seeding torrents though...
Okay I'll give it a shot..
Just a side-note, I think it may need to only apply when a single file is downloaded, because if LL makes a download with more than 1 file, I believe it automatically sets up that sub-dir in the downloads directory.
It depends on the uploader. Lots of torrents and almost all nzb contain a folder with the file(s) in. Problem is if we move the downloaded file(s), what happens if you want to keep seeding? The torrent programs will complain, I think.
Maybe instead of move/copy we can use a Hardlink (like sonarr)?
IS there a setting to test out the 'move' for now or is it automatically doing move if "keep original files" is unchecked?
Updated to 9b5daf82e4f1a54fa192f671e79acb42e4330f7e and still seems to be copying
Actually I have just re-read your earlier message and I think I misremembered how lazylibrarian currently works. I had a quick look at the code, it seems this is what happens...
When LL finds a download with only 1 book it downloads only that file to the download dir - and does >not create a sub directory: /downloads/bookname.epub This is the single file downloaded, no directory provided by the uploader
Once the post processing runs, the sub-dir gets set up correctly, downloads the metadata and copies >the book there, but it keeps the original book file in the main download dir for seeding: /downloads/bookname.epub /downloads/book name/bookname.epub /downloads/book name/bookname.opf /downloads/book name/bookname.jpg This subdir is created by lazylibrarian, and the opf and jpg are also created by lazylibrarian
I think if the download was a torrent/magnet the original single file should be kept for seeding. The lazylibrarian created subdir and all it's folders should be moved into autoadd, not copied. Current version does this.
Hard links are a good idea, but I don't know if they work on windows/mac, and across filesystems?
I dont think the current version is doing this correctly - or at least its not on mine.
LL is correctly keeping the single file for seeding, and when its done in transmission it deletes that single file from the main download dir, however the sub-dir that is created with the book file that is then moved is not actually being moved - its still being copied.
After all seeding is complete, I have the book in my calibre autoadd dir, but still have that sub dir in my download dir with the book file inside of it.
In regards to syslinks, I dont think they work across filesystems, although each OS is capable (not 100% sure on this one though)
UPDATE: A few more single-file books download last night and LL is still not doing a copy instead of move.
Would using the same dir for "Book Library Folder" as my calibre library be what could be causing this issue?
/books is my main dir that has all of the calibre generated folders with random numbers on book titles in it "i.e. /books/Stephen King/Book Name (XX)"
Right now I have the following settings: "Book Library Folder" /books "eBook Foldername Pattern" /books "eBook Filename Pattern" $Title - $Author "calibredb import program" blank "Calibre Auto Add Directory" blank
Since Calibre generates those numbers at the end of the folder names I left the Foldername Pattern as /books... It seems to still be able to pick up where the books are in these sub directories as downloading a book from the LL browser works by clicking on "Open".
When post processing happens, what exactly does LL try to do? Right now it is both creating a sub-dir in my downloads folder (in addition to the file that is still seeding) as well as placing a copy in the /books folder, even when I have the Calibre Auto Add Directory setting blank.
I have a feeling it is trying to place the sub dir that it creates in the download dir into the /book dir and my settings above may be throwing it off... not sure if that is exactly how the post process works though.
Thanks
Sharing the calibre library folder is fine, but works best if you use calibredb for the reason you have noticed, calibre adds a number to the end of the book foldername. It uses that as it's own internal identifier, and you only get to know the number if you use calibredb. With calibredb we get to know exactly where calibre put the book so we can link to it. Kovid (calibre author) has been asked to change this many times, but he won't. There is really no need to put the id number in the foldername, it's already in the calibre database and the calibre generated opf files, and modifying the foldername just makes it difficult for 3rd party programs. There is a similar issue with renaming authors or book titles in the calibre program, all the folder names move too.
If not using calibredb, auto-add is the next best option. LazyLibrarian will postprocess into a temporary directory, and then scan the calibre library looking for author directory, and scan that looking for any new books.
When postprocessing, if the downloaded torrent/nzb is a single file lazylibrarian will first create a sub-folder using the filename, and copy the book into that directory. This is so that we leave the seeding copy alone, and if we need to create opf and cover images they are kept in a folder with the book.
If you are not using calibre auto-add or calibredb then lazylibrarian will copy/move this new folder (the one we created or the downloaded folder) into the lazylibrarian library using the eBook folder name pattern. Usually author_dir with each book it it's own sub-dir. The assumption is that if you are not using calibre auto-add and not using calibredb, you aren't using calibre at all, so we move it ourselves using our own naming scheme. We move all configured book formats, plus opf, plus jpg, renaming them so they all start with the book title.
If you use /books for both calibre and lazylibrarian, using the default naming pattern for lazylibrarian ebook folder will mean you end up with some book folders with calibre numbers, some without. Those with a number are the only ones calibre knows about, lazylibrarian knows about both. When you next update your calibre library, it will import the ones without a number, and rename their folders adding its own number to the end. After that lazylibrarian can't find them, but if you use lazylibrarian libraryscan it will rescan the calibre library and find them again including the number. Not perfect, but that's why calibredb is the preferred option.
Ah okay thanks for laying that out... an idea:
Would it be possible to set up an option for when using Auto-Add to just move the .epub file that is downloaded to a specific dir as opposed to creating the sub dir? ...And if possible ignore the .jpg and .opf's?
It seems like really all Calibre needs is the book file placed in its dir, and with auto-add turned on in calibre, it can automatically download the cover/metadata and set up the proper directories.
Workflow would look something like this:
Right now it seems like the subdir is getting created in the download folder, everything is getting copied over to calibre autoadd dir, and then calibre just detects the book file to import, so its sets up all of its dir's and re-downloads the cover and metadata because it doesn't pull those in from the auto-add feature by just placing those files in the directory (Doesnt read .opf/.jpg).
I've tried the calibredb approach but its rather complex since both LL and Calibre are running in their own docker containers.
I think auto-add should move files if "keep original files" is not set, would have to check. We include the opf and jpg in the auto add as if calibre is set to import using file contents it can use them. Another thought, can you copy calibredb into the /books folder, since lazylibrarian can see that? You should be able to put /books/calibredb as your calibredb path in lazylibrarian config. Actually I just had a quick look, it's just a wrapper so you would need the whole calibre installation.
"Keep original files" currently is unchecked and it's still doing the copy instead of move.
The .opf/.jpg seem to be getting copied over but then once in calibre dir it looks like it's only doing a "move" on the epub, because I have the opfs/jpgs sitting in the main dir as well as in the sub dirs calibre creates - however the epub gets moved correctly and is not duplicated in the auto add dir
The main concern with copying calibredb into LL is that I do not think it will run correctly. Each program is in its own docker (virtual environment) so the actually calibre program doesn't exist in LL world and LL doesn't exist in calibre world... Will try playing with this further as I may be able to get to the desired result by linking the containers - ideally I'd prefer auto add approach though because linking the 2 would force me to make both LL and calibre accessible to external network whereas right now only calibre is for its content server
ok, I will take a look at the copy/move section tomorrow. It may be calibre reads the opf, maybe not, depending on it's settings as I said before.
I think if you have two versions of calibre, one inside the lazylibrarian docker and one in it's own docker, only the standalone one will be exposed to outside. The one inside lazylibrarian docker will only be used by lazylibrarian (actually only the calibredb part) because the calibredb inside the lazylibrarian docker only exists inside that docker, in the same way the private lazylibrarian version of ghostscript only exists for lazylibrarian.
I have asked linuxserver.io for clarification, they are usually very helpful.
I had a quick look on calibres site and it seems you can install it with a single command... sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.py | sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
@philborman is correct, if calibredb is visible to the docker container and appropriate permissions and dependencies are satisfied there's no reason it can't run.
Hang on, are we talking about a binary component of calibre?
If so: -v /lazylibrarian/calibredb:/usr/sbin/calibredb
ie mount the file in the same manner as a folder is mounted in docker.
You of course would need to get the calibredb binary from somewhere, as well as satisfy all the dependencies etc as I mentioned before.
@CHBMB I think that mount command you shared may solve all of this - will test as soon as I'm home!
@CHBMB It looks like there is a docker that already does this? https://hub.docker.com/r/thraxis/lazylibrarian-calibre/
Is this a one of the linuxserver.io stacks? The config parameters look identical...
I know aptalca calibre-rdp (calibre docker I'm running) isn't "technically" on the linuxserver.io repo but the dev works very closely with you guys...
No, nothing to do with us, guy just ripped off our logo and stuff. aptalca is one of our members but still has some solitary repos, as does sparklyballs.
If that works for you that's great, but I need to make it perfectly clear we won't support it, you have to contact that docker author.
Ahh that's a shame - Unfortunately I do not have the skillset to confidently customize the linuxserver.io docker to add the binary to it, otherwise I would. I'm not crazy about the fact that its a rip-off with no props given to the linuxserver team, but I think this may have to be the route I go given that it satisfies my need and I'm not sure of any other relatively straight-forward options... Thanks for all the awesome other containers though! :D
@philborman I did just test this out and confirm it solves my issues mentioned above - no duplicates anywhere... I did however notice a few other downsides...
Could you please confirm if the following behavior is expected of calibredb import program, or perhaps an issue with the thraxis docker? - I realize this may be more of a calibre question than LL at this point but appreciate any info you can share if you are knowledgeable of it:
Although the calibre directories have been set up correctly, calibre does not see the added books until I restart/refresh it
With the auto-add, calibre was able to auto convert to the output format (for me .mobi). I have this option selected in both the "Adding actions" and "Automatic Adding" settings in calibre but it appears to not be working even after the refresh
Thanks again in advance for all your help ...almost there!
Okay I'll give the example_ebook_convert.py script a go... Here's my logs:
Timestamp | Level | Message |
---|---|---|
2017-07-22 15:43:33 | DEBUG | Restart processDir job in 10 minutes |
2017-07-22 15:43:33 | DEBUG | Stop processDir job |
2017-07-22 15:43:33 | INFO | 1 book/mag processed. |
2017-07-22 15:43:33 | DEBUG | Snatched direct Amy Poehler - Yes Please LL.(20910157) is not in download directory |
2017-07-22 15:43:33 | DEBUG | /downloads/Amy Poehler - Yes Please LL.(20910157) is not a directory? |
2017-07-22 15:43:33 | INFO | Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads |
2017-07-22 15:43:33 | DEBUG | Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT |
2017-07-22 15:43:33 | INFO | Successfully processed: Yes Please - Amy Poehler |
2017-07-22 15:43:33 | DEBUG | Deleted Amy Poehler - Yes Please LL.(20910157), direct from direct |
2017-07-22 15:43:33 | ERROR | processExtras: No dest_file supplied |
2017-07-22 15:43:33 | DEBUG | Processed direct: Yes Please - Amy Poehler, http://download.libgen.io/get.php?md5=4515217CF5F7C85B8DA12383618EC287&key=BWBNKBGAXWSKNCOJ |
2017-07-22 15:43:33 | INFO | Library scan complete |
2017-07-22 15:43:33 | DEBUG | Updating bookcounts for 1 author |
2017-07-22 15:43:33 | INFO | 2 files processed |
2017-07-22 15:43:33 | INFO | 1/1 new/modified eBooks found and added to the database |
2017-07-22 15:43:33 | DEBUG | Amy Poehler Yes Please (9780062268372) matched Snatched BookID 20910157, [Amy Poehler][Yes Please] |
2017-07-22 15:43:33 | WARNING | Updating book location for Amy Poehler Yes Please (9780062268372) from /books/Amy Poehler/Yes Please (118)/Yes Please - Amy Poehler.epub to /books/Amy Poehler/Yes Please (119)/Yes Please - Amy Poehler.epub |
2017-07-22 15:43:33 | DEBUG | Link to preferred type epub: /books/Amy Poehler/Yes Please (119)/Yes Please - Amy Poehler.epub |
2017-07-22 15:43:33 | DEBUG | Fuzz match partial [99] [Yes Please (9780062268372)] [Yes Please] |
2017-07-22 15:43:33 | DEBUG | Searching 1 book by [Amy Poehler] in database for [Yes Please (9780062268372)] |
2017-07-22 15:43:33 | DEBUG | Searching database for [Yes Please (9780062268372)] by [Amy Poehler] |
2017-07-22 15:43:33 | DEBUG | Preferred authorname changed from [Poehler, Amy] to [Amy Poehler] |
2017-07-22 15:43:33 | DEBUG | Formatted authorname [Poehler, Amy] to [Amy Poehler] |
2017-07-22 15:43:33 | DEBUG | book meta [] [en] [Poehler, Amy] [Yes Please (9780062268372)] [epub] |
2017-07-22 15:43:33 | DEBUG | [/books/Amy Poehler/] Now scanning subdirectory Yes Please (119) |
2017-07-22 15:43:33 | INFO | Scanning eBook directory: /books/Amy Poehler/ |
2017-07-22 15:43:33 | DEBUG | /opt/calibre/calibredb identifier reports: No write acces to /root/.config/calibre using a temporary dir instead Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-22T19:43:27+00:00 Published : 0101-01-01T00:00:00+00:00 Identifiers : goodreads:20910157 |
2017-07-22 15:43:32 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'identifiers:goodreads:20910157', '--with-library', u'/books', '119'] |
2017-07-22 15:43:32 | DEBUG | /opt/calibre/calibredb title reports: No write acces to /root/.config/calibre using a temporary dir instead Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-22T19:43:27+00:00 Published : 0101-01-01T00:00:00+00:00 |
2017-07-22 15:43:31 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'title:Yes Please', '--with-library', u'/books', '119'] |
2017-07-22 15:43:31 | DEBUG | /opt/calibre/calibredb author reports: No write acces to /root/.config/calibre using a temporary dir instead Title : Yes Please (9780062268372) Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-22T19:43:27+00:00 Published : 0101-01-01T00:00:00+00:00 |
2017-07-22 15:43:29 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'authors:Amy Poehler', '--with-library', u'/books', '119'] |
2017-07-22 15:43:29 | DEBUG | Calibre ID: 119 |
2017-07-22 15:43:29 | DEBUG | /opt/calibre/calibredb reports: No write acces to /root/.config/calibre using a temporary dir instead Added book ids: 119 |
2017-07-22 15:43:26 | DEBUG | [u'/opt/calibre/calibredb', 'add', '-1', u'--with-library=/books', u'/downloads/Amy Poehler - Yes Please LL.(20910157)'] |
2017-07-22 15:43:26 | DEBUG | Importing Yes Please - Amy Poehler into calibre library |
2017-07-22 15:43:26 | DEBUG | Processing eBook 20910157 |
2017-07-22 15:43:26 | DEBUG | Found match (100%): /downloads/Amy Poehler - Yes Please LL.(20910157) for eBook Amy Poehler - Yes Please LL.(20910157) |
2017-07-22 15:43:26 | DEBUG | Found folder (100%) [/downloads/Amy Poehler - Yes Please LL.(20910157)] for eBook Amy Poehler - Yes Please |
2017-07-22 15:43:26 | INFO | Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads |
2017-07-22 15:43:26 | DEBUG | Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT |
2017-07-22 15:43:26 | DEBUG | Found 2 files marked "Snatched" |
2017-07-22 15:43:26 | DEBUG | Checking 1 file in /downloads |
2017-07-22 15:43:23 | INFO | Log cleared, level set to [2]- Log Directory is [/config/log] |
No write acces to /root/.config/calibre Looks like you need to make it rw ?
I saw those in the DEBUG but was not sure if that was directly causing the "processExtras: No dest_file supplied" error...
I believe the /root/.config/calibre directory is on the container side, and isn't mounted through the /config volume, so every time the image is pulled to refresh this would have to manually be updated - I opened an issue on the docker specific page as this permission should probably be set by default: https://github.com/Thraxis/docker-lazylibrarian-calibre/issues/2
On a side note - I tried SSHing into the docker and couldn't find this dir manually...
In terms of the notifier - I am assuming it is just not working because of this issue above, however I've downloaded the script, updated the ebook-converter full path, set the permissions to rwxr-xr-- and updated the LL settings...
Hoping to get a quick response there as it should be a quick fix for the dev to just apply the write permissions.
It should be fixed now
A few more thoughts. The "processExtras: No dest_file supplied" error... is because we think calibre failed to import the book properly, and the notifier isn't running for the same reason, no new book to notify you about.
It looks like calibredb talks to a running calibre instance through /root/.config/calibre so I think you probably need to run the thraxis container for both calibre and lazylibrarian and stop the standalone calibre one. That could also be why you are having to restart/refresh calibre after adding books, the "wrong" calibre is being told about changes.
@philborman - Tested this again with calibre shutdown, still getting the "processExtras: No dest_file supplied" error
I've just reworked the calibredb section, more progress messages on debug, and a different way of getting dest_file (work it out ourselves instead of asking calibre). Might solve the problem?
Still on the Thraxis docker (re-pulled image) but not sure if your changes made it through yet...
I did do a full library scan and author refresh and don't seem to be getting the "processExtras: No dest_file supplied" error anymore...
...Now this though:
Timestamp | Level | Message |
---|---|---|
2017-07-23 12:44:41 | ERROR | Unhandled exception in processDir: Traceback (most recent call last): File "/app/lazylibrarian/lazylibrarian/postprocess.py", line 459, in processDir global_name, book['BookID'], book_type) File "/app/lazylibrarian/lazylibrarian/postprocess.py", line 1044, in processDestination imported = LibraryScan(calibre_dir, remove=remove) # rescan whole authors directory UnboundLocalError: local variable 'remove' referenced before assignment |
2017-07-23 12:44:41 | DEBUG | Calibre trying directory [/books/Amy Poehler/Yes Please - Amy Poehler (210)] |
2017-07-23 12:44:41 | DEBUG | /opt/calibre/calibredb identifier reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T16:44:35+00:00 Published : 0101-01-01T00:00:00+00:00 Identifiers : goodreads:20910157 |
2017-07-23 12:44:40 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'identifiers:goodreads:20910157', '--with-library', u'/books', '210'] |
2017-07-23 12:44:40 | DEBUG | /opt/calibre/calibredb title reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T16:44:35+00:00 Published : 0101-01-01T00:00:00+00:00 |
2017-07-23 12:44:39 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'title:Yes Please', '--with-library', u'/books', '210'] |
2017-07-23 12:44:39 | DEBUG | /opt/calibre/calibredb author reports: Title : Yes Please (9780062268372) Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T16:44:35+00:00 Published : 0101-01-01T00:00:00+00:00 |
2017-07-23 12:44:37 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'authors:Amy Poehler', '--with-library', u'/books', '210'] |
2017-07-23 12:44:37 | DEBUG | Calibre ID: 210 |
2017-07-23 12:44:37 | DEBUG | /opt/calibre/calibredb reports: Added book ids: 210 |
2017-07-23 12:44:35 | DEBUG | [u'/opt/calibre/calibredb', 'add', '-1', u'--with-library=/books', u'/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)'] |
2017-07-23 12:44:35 | DEBUG | Importing Yes Please - Amy Poehler into calibre library |
2017-07-23 12:44:35 | DEBUG | Processing eBook 20910157 |
2017-07-23 12:44:35 | DEBUG | Found match (100%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) for eBook Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 12:44:35 | DEBUG | Found folder (100%) [/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)] for eBook Amy Poehler - Yes Please |
2017-07-23 12:44:35 | INFO | Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:35 | DEBUG | Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT |
2017-07-23 12:44:35 | DEBUG | Closest match (26%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 12:44:35 | DEBUG | Snatched direct David Baldacci - Memory Man LL.(23153154) is not in download directory |
2017-07-23 12:44:35 | INFO | Looking for eBook David Baldacci - Memory Man LL.(23153154) in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:35 | DEBUG | David Baldacci - Memory Man LL.(23153154) was sent to DIRECT |
2017-07-23 12:44:35 | DEBUG | Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 12:44:35 | DEBUG | Snatched torznab Tarryn Fisher - Marrow LL.(25379213) is not in download directory |
2017-07-23 12:44:35 | INFO | Looking for eBook Tarryn Fisher - Marrow LL.(25379213) in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:35 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 12:44:35 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Tarryn Fisher - Marrow LL.(25379213) was sent to TRANSMISSION |
2017-07-23 12:44:35 | DEBUG | Closest match (18%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 12:44:35 | DEBUG | Snatched torznab Megan Miranda - All the Missing Girls LL.(27276324) is not in download directory |
2017-07-23 12:44:35 | INFO | Looking for eBook Megan Miranda - All the Missing Girls LL.(27276324) in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:35 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 12:44:35 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Megan Miranda - All the Missing Girls LL.(27276324) was sent to TRANSMISSION |
2017-07-23 12:44:35 | DEBUG | Closest match (42%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 12:44:35 | DEBUG | Snatched torznab Paulo Coelho - The Alchemist LL.(865) is not in download directory |
2017-07-23 12:44:35 | INFO | Looking for eBook Paulo Coelho - The Alchemist LL.(865) in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:35 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 12:44:35 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Paulo Coelho - The Alchemist LL.(865) was sent to TRANSMISSION |
2017-07-23 12:44:35 | DEBUG | Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 12:44:35 | DEBUG | Snatched magnet Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK is not in download directory |
2017-07-23 12:44:35 | INFO | Looking for eBook Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:35 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 12:44:35 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 12:44:35 | DEBUG | Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK was sent to TRANSMISSION |
2017-07-23 12:44:35 | DEBUG | Found 6 files marked "Snatched" |
2017-07-23 12:44:35 | DEBUG | Checking 1 file in /downloads/completed/LazyLibrarian |
2017-07-23 12:44:33 | INFO | Log cleared, level set to [2]- Log Directory is [/config/log] |
ah- just fixed that. New version out shortly
okay great - I also noticed the new functionality you added for "Autoadd eBook only", thanks for getting this added!
1 thought around it - would it be possible to have this also apply when not using either of the "Calibre Auto Add Directory" or "Calibredb import program" functionalities?
By setting the "eBook Foldername Pattern" the same as the "Book Library Folder", LL will currently move all of the downloaded files directly to the main /book directory, which Calibre monitors only for the book files to work its auto-add magic...
By doing this, we can remove LL from any of the calibre processing - and on the next library scan, LL will see the books in the dirs that calibre automatically sets up.
If we can set LL to only send the .epub over then there wont be any duplicates .opf/.jpgs leftover in the /book dir. Similarly, there wont be any duplicate epubs left over in the download dir as they are being moved to /book and not copied.
Just tested out calibredb on the new version.
Everything seems to be getting placed in the correct places, only problem is calibre still doesn't show the book until the library is manually refreshed or calibre is restarted.
2017-07-23 19:10:52 | DEBUG | Restart processDir job in 10 minutes |
---|---|---|
2017-07-23 19:10:52 | DEBUG | Stop processDir job |
2017-07-23 19:10:52 | INFO | 1 book/mag processed. |
2017-07-23 19:10:52 | DEBUG | Snatched direct Amy Poehler - Yes Please LL.(20910157) is not in download directory |
2017-07-23 19:10:52 | DEBUG | /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) is not a directory? |
2017-07-23 19:10:52 | INFO | Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:52 | DEBUG | Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT |
2017-07-23 19:10:51 | DEBUG | Pushbullet devices: None |
2017-07-23 19:10:51 | DEBUG | Pushbullet api: XXXXXXXXXXXXXXXXXXXX |
2017-07-23 19:10:51 | DEBUG | Pushbullet message: eBook Yes Please - Amy Poehler from libgen.io/search.php at 2017-07-23 19:10:51 |
2017-07-23 19:10:51 | DEBUG | Pushbullet event: Added to Library |
2017-07-23 19:10:51 | DEBUG | Pushbullet: Sending notification eBook Yes Please - Amy Poehler from libgen.io/search.php at 2017-07-23 19:10:51 |
2017-07-23 19:10:51 | INFO | Successfully processed: Yes Please - Amy Poehler |
2017-07-23 19:10:51 | DEBUG | Deleted Amy Poehler - Yes Please LL.(20910157), direct from direct |
2017-07-23 19:10:51 | DEBUG | Calibre should have created the extras for us |
2017-07-23 19:10:51 | DEBUG | Processed direct: Yes Please - Amy Poehler, http://download.libgen.io/get.php?md5=4515217CF5F7C85B8DA12383618EC287&key=JBXFJ1JUJOBJO4Y3 |
2017-07-23 19:10:51 | INFO | Library scan complete |
2017-07-23 19:10:51 | DEBUG | Updating bookcounts for 1 author |
2017-07-23 19:10:51 | INFO | 2 files processed |
2017-07-23 19:10:51 | INFO | 1/1 new/modified eBooks found and added to the database |
2017-07-23 19:10:51 | DEBUG | Amy Poehler Yes Please (9780062268372) matched Snatched BookID 20910157, [Amy Poehler][Yes Please] |
2017-07-23 19:10:51 | WARNING | Updating book location for Amy Poehler Yes Please (9780062268372) from /books/Amy Poehler/Yes Please (186)/Yes Please - Amy Poehler.epub to /books/Amy Poehler/Yes Please (226)/Yes Please - Amy Poehler.epub |
2017-07-23 19:10:51 | DEBUG | Librarysync link to preferred type epub: /books/Amy Poehler/Yes Please (226)/Yes Please - Amy Poehler.epub |
2017-07-23 19:10:50 | DEBUG | Fuzz match partial [99] [Yes Please (9780062268372)] [Yes Please] |
2017-07-23 19:10:50 | DEBUG | Searching 1 book by [Amy Poehler] in database for [Yes Please (9780062268372)] |
2017-07-23 19:10:50 | DEBUG | Searching database for [Yes Please (9780062268372)] by [Amy Poehler] |
2017-07-23 19:10:50 | DEBUG | Preferred authorname changed from [Poehler, Amy] to [Amy Poehler] |
2017-07-23 19:10:50 | DEBUG | Formatted authorname [Poehler, Amy] to [Amy Poehler] |
2017-07-23 19:10:50 | DEBUG | book meta [] [en] [Poehler, Amy] [Yes Please (9780062268372)] [epub] |
2017-07-23 19:10:50 | DEBUG | [/books/Amy Poehler/] Now scanning subdirectory Yes Please (226) |
2017-07-23 19:10:50 | INFO | Scanning eBook directory: /books/Amy Poehler/ |
2017-07-23 19:10:50 | DEBUG | Calibre trying directory [/books/Amy Poehler/Yes Please - Amy Poehler (226)] |
2017-07-23 19:10:50 | DEBUG | /opt/calibre/calibredb identifier reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T23:10:44+00:00 Published : 0101-01-01T00:00:00+00:00 Identifiers : goodreads:20910157 |
2017-07-23 19:10:49 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'identifiers:goodreads:20910157', '--with-library', u'/books', '226'] |
2017-07-23 19:10:49 | DEBUG | /opt/calibre/calibredb title reports: Title : Yes Please Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T23:10:44+00:00 Published : 0101-01-01T00:00:00+00:00 |
2017-07-23 19:10:48 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'title:Yes Please', '--with-library', u'/books', '226'] |
2017-07-23 19:10:48 | DEBUG | /opt/calibre/calibredb author reports: Title : Yes Please (9780062268372) Title sort : Yes Please (9780062268372) Author(s) : Amy Poehler [Poehler, Amy] Publisher : HarperCollins Languages : eng Timestamp : 2017-07-23T23:10:44+00:00 Published : 0101-01-01T00:00:00+00:00 |
2017-07-23 19:10:46 | DEBUG | [u'/opt/calibre/calibredb', 'set_metadata', '--field', u'authors:Amy Poehler', '--with-library', u'/books', '226'] |
2017-07-23 19:10:46 | DEBUG | Calibre ID: 226 |
2017-07-23 19:10:46 | DEBUG | /opt/calibre/calibredb reports: Added book ids: 226 |
2017-07-23 19:10:43 | DEBUG | [u'/opt/calibre/calibredb', 'add', '-1', u'--with-library=/books', u'/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)'] |
2017-07-23 19:10:43 | DEBUG | Importing Yes Please - Amy Poehler into calibre library |
2017-07-23 19:10:43 | DEBUG | Processing eBook 20910157 |
2017-07-23 19:10:43 | DEBUG | Found match (100%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) for eBook Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 19:10:43 | DEBUG | Found folder (100%) [/downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157)] for eBook Amy Poehler - Yes Please |
2017-07-23 19:10:43 | INFO | Looking for eBook Amy Poehler - Yes Please LL.(20910157) in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:43 | DEBUG | Amy Poehler - Yes Please LL.(20910157) was sent to DIRECT |
2017-07-23 19:10:43 | DEBUG | Closest match (26%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 19:10:43 | DEBUG | Snatched direct David Baldacci - Memory Man LL.(23153154) is not in download directory |
2017-07-23 19:10:43 | INFO | Looking for eBook David Baldacci - Memory Man LL.(23153154) in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:43 | DEBUG | David Baldacci - Memory Man LL.(23153154) was sent to DIRECT |
2017-07-23 19:10:43 | DEBUG | Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 19:10:43 | DEBUG | Snatched torznab Tarryn Fisher - Marrow LL.(25379213) is not in download directory |
2017-07-23 19:10:43 | INFO | Looking for eBook Tarryn Fisher - Marrow LL.(25379213) in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:43 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:43 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:38 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:38 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Tarryn Fisher - Marrow LL.(25379213) was sent to TRANSMISSION |
2017-07-23 19:10:33 | DEBUG | Closest match (18%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 19:10:33 | DEBUG | Snatched torznab Megan Miranda - All the Missing Girls LL.(27276324) is not in download directory |
2017-07-23 19:10:33 | INFO | Looking for eBook Megan Miranda - All the Missing Girls LL.(27276324) in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:33 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 19:10:33 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Megan Miranda - All the Missing Girls LL.(27276324) was sent to TRANSMISSION |
2017-07-23 19:10:33 | DEBUG | Closest match (42%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 19:10:33 | DEBUG | Snatched torznab Paulo Coelho - The Alchemist LL.(865) is not in download directory |
2017-07-23 19:10:33 | INFO | Looking for eBook Paulo Coelho - The Alchemist LL.(865) in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:33 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 19:10:33 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Paulo Coelho - The Alchemist LL.(865) was sent to TRANSMISSION |
2017-07-23 19:10:33 | DEBUG | Closest match (29%): /downloads/completed/LazyLibrarian/Amy Poehler - Yes Please LL.(20910157) |
2017-07-23 19:10:33 | DEBUG | Snatched magnet Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK is not in download directory |
2017-07-23 19:10:33 | INFO | Looking for eBook Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:33 | DEBUG | getTorrentFolder: No response from transmission |
2017-07-23 19:10:33 | DEBUG | Requesting URL via POST method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Requesting URL via GET method: http://10.0.3.1:9091/transmission/rpc |
2017-07-23 19:10:33 | DEBUG | Everything All at Once by Bill Nye ePUB MOBI eBOOK-ZAK was sent to TRANSMISSION |
2017-07-23 19:10:33 | DEBUG | Found 7 files marked "Snatched" |
2017-07-23 19:10:33 | DEBUG | Checking 1 file in /downloads/completed/LazyLibrarian |
2017-07-23 19:10:29 | INFO | Log cleared, level set to [2]- Log Directory is [/config/log] |
Yes that would be the correct behavior currently.
Ah okay thanks for clarifying... Is that just for Calibre?
For example if I were running COPS or Calibre-Web, would those have to be restarted as well for the books to show?
I just imported a book and COPS did after a little bit display it in the UI. but that could have been due to my changing from the default template to the bootstrap one
Any way to pass a "refresh" command through calibredb after a book is added to be sure?
There doesn't seem to be a separate command to do this, it should happen automatically with each calibredb command, see log below. After each command you get a message from calibredb "Notifying calibre of the change". I guess if you are not seeing this message, calibredb doesn't know calibre is running, or can't talk to it. I don't know how it checks, but I have calibre running on the same machine as calibredb, so they must communicate somehow, and in a way that doesn't need any configuring since I have never done any. Maybe a socket? Maybe it looks for the pid of calibre and sends a signal?
There is another way of talking to a calibre content server with calibredb, using http://host:port/#library_id which in theory lets calibredb talk to calibre on another machine (or in another docker) but I haven't managed to get that to work. The documentation says http://host:port/#- should give a list of library ids but I just get "Not Found"
24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : Found match (100%): /media/Popcorn/Download/complete/Books/Prince Lestat and the Realms of Atlantis LL.(23267873) for eBook Prince Lestat and the Realms of Atlantis 24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : Processing eBook 23267873 24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : Importing Prince Lestat and the Realms of Atlantis - Anne Rice into calibre library 24-Jul-2017 12:47:13 - DEBUG :: POSTPROCESS : [u'calibredb', 'add', '-1', u'--with-library=/media/Popcorn/Calibre Library/', u'/media/Popcorn/Download/complete/Books/Prince Lestat and the Realms of Atlantis LL.(23267873)'] 24-Jul-2017 12:47:32 - DEBUG :: POSTPROCESS : calibredb reports: Backing up metadata Added book ids: 1851 Notifying calibre of the change
24-Jul-2017 12:47:32 - DEBUG :: POSTPROCESS : Calibre ID: 1851
24-Jul-2017 12:47:32 - DEBUG :: POSTPROCESS : [u'calibredb', 'set_metadata', '--field', u'authors:Anne Rice', '--with-library', u'/media/Popcorn/Calibre Library/', '1851']
24-Jul-2017 12:47:44 - DEBUG :: POSTPROCESS : calibredb author reports: Title : Prince Lestat and the Realms of Atlantis
Title sort : Prince Lestat and the Realms of Atlantis
Author(s) : Anne Rice [Rice, Anne]
Publisher : Knopf Doubleday Publishing Group
Languages : eng
Timestamp : 2017-07-24T10:47:22+00:00
Published : 2016-11-28T23:00:00+00:00
Comments : From Anne Rice, conjurer of the beloved best sellers Interview with a Vampire and Prince Lestat, an ambitious and exhilarating new novel of utopian vision and power, uniting the ancient worlds and the legends of the Vampire Chronicles.
"In my dreams, I saw a city fall into the sea. I heard the cries of thousands. I saw flames that outshone the lamps of heaven. And all the world was shaken . . ." —Anne Rice, Prince Lestat and the Realms of Atlantis
At the novel's center: the vampire Lestat de Lioncourt, hero, leader, inspirer, irresistible force, irrepressible spirit, battling (and ultimately reconciling with) a strange otherworldly form that has somehow taken possession of Lestat's undead body and soul. This ancient and mysterious power and unearthly spirit of vampire lore has all the force, history, and insidious reach of the unknowable Universe.
It is through this spirit, previously considered benign for thousands of vampire...
Backing up metadata
Notifying calibre of the change
The instructions are a little unclear on how the remote content server works with calibredb.
First of all you must have the Calibre Content Server running. Just having the GUI running does not start the Content Server. Click on Connect/Share and Start the Content Server. Note the IP address and port.
The command-line for calibredb would then be:
/opt/calibre/calibredb --with-library="http://
Which will give you the list of library names, something like as follows: FanFiction FanFiction_(Complete) SciFi
/opt/calibre/calibredb --with-library="http://
I'm not sure how usefull this would actually be as there is no "refresh" option and the remote commands only seem to allow you to control a remote Content Server, which means that both the system LL is on and the Calibre Content Server are on need to have access to the path where the book to be added is located, there is no provision for "uploading" the file to the Content Server.
So you pretty much either have to run dockers with identical paths, or you have to have the path mapping for the remote system.
I have a running calibre-server on port 9000 and can talk to it using a web browser, but the calibredb commands just don't work for me. calibredb --with-library="http://localhost:9000/#-" list gives the Usage page
calibredb list --with-library="http://localhost:9000/#-" id title authors (as if it's trying to list the library contents but can't find any, not listing the library ids)
calibredb list --with-library="http://localhost:9000/#Calibre Library" id title authors (same)
calibredb list --with-library="/home/phil/Calibre Library" lists all the books
@philborman - I pulled the update for the "Autoadd eBook only" but it still looks like when not using the claibre autoadd or calibredb the extra files are still being moved to the main book dir.
I did try that way, but there are problems because the "main book dir" is your lazylibrarian library. If you don't put a cover image in the library, lazylibrarian will download a new one, as it needs the cover for it's own display purposes. We could miss out the opf file, as at the moment lazylibrarian doesn't use it.
Ah I see - what is interesting here though is that when calibre does its own "Auto Add" by seeing the .epub in /books, that .jpg is somehow making it into the new dir it sets up.
I am not sure if calibre is using the .jpg that LL downloaded and copying it while moving the .epub to the new dir, or if it is downloading the .jpg on its own once it moves the .epub... Either way the .jpg is making its way into the dir that claibre sets up, and LL can read that .jpg because they still show in LL after I delete those .jpgs from the main /book dir and do a library scan.
Could be either calibre copying it or downloading it's own, hard to say, depends on calibre settings, among other things. Easy way to find out is to deliberately put the wrong image in there and see if calibre picks it up. Calibre will also read our opf file, again depending on calibre settings, and also depending on embedded book data - seems calibre prefers embedded data to opf data, but will read other fields from the opf if they are not present in embedded data. Also our opf includes a link to the book image, so calibre could pick it up from there.
docker linux, QNAP
Current Version: 4f2f8857678ab423cd2831e6a6924b89ec5c5a7a
I am getting the following error when trying to use the calibredb import program: calibredb failed, Text file busy
Book dir and calibre metadata.db both have 777...
Log snippet: