search

DobyTang / LazyLibrarian

This project isn't finished yet. Goal is to create a SickBeard, CouchPotato, Headphones-like application for ebooks. Headphones is used as a base, so there are still a lot of references to it.
732 stars 71 forks source link

Auto Importing to Calibre #1040

Closed mccorkled closed 6 years ago

mccorkled commented 7 years ago

I have SAB downloading the books. LL post processes to /books. How do I go about having LL auto add the books to the Calibre db? I do not have an existing one.

Thank you.

philborman commented 7 years ago

Are you already using calibre? There are several ways to make calibre and lazylibrarian share books, but which way you use depends on what you are trying to do. If you are not using calibre, be aware that when you start using it, it will force a rename of all the books and folders to it's own preferred pattern, and lazylibrarian will have to rescan everything.

mccorkled commented 7 years ago

I was trying to use Calibre-web initially but the developer didn't want to add an auto add book feature. I am now using Calibre and it's running but I do not have a db.

philborman commented 7 years ago

We can only talk to an existing calibre database, can't create a new one.

mccorkled commented 7 years ago

Do you know of a way (I am new to all of this) to get my books to auto add to a directory like sonarr/radarr does for tv/movies? I cannot have Calibre running on a local machine. Everything needs to be on the server.

philborman commented 7 years ago

See the wiki for calibre integration details, there are lots of options depending on what you want to do. https://github.com/DobyTang/LazyLibrarian/wiki/Configuration

mccorkled commented 7 years ago

I have read this and I still am unsure of how to make it work.

philborman commented 6 years ago

Difficult to give a better answer without more info. Are you running calibre and lazylibrarian on the same machine What version of calibre (v2.x and v3.x have different options) You said in an earlier post that you have calibre running but no db, but lazylibrarian cannot create a new calibre.db. We can only add books to an existing db.

mccorkled commented 6 years ago

Both Calibre and LL are running on my NAS. Calibre 3.1.1 and I got the db working. Thank you.

philborman commented 6 years ago

That's about the simplest set up to get working, both servers on the same machine, calibre v3.x, no authentication needed. What I have in lazylibrarian config is...

Calibredb import program:  calibredb
Use Calibre Content Server: <ticked>
Calibre Server: http://127.0.0.1:9000
Calibre Username: <nothing>
Calibre Password: <nothing>
Calibre Auto Add Directory: <nothing>

In calibre itself, go to preferences, sharing over the net, and check the port is set to 9000 (I think that's the default). Untick "require username" and tick "run server automatically", then click "Apply" at the bottom of the page. To start the server, click the Connect/share button and choose Start Content server.

Once it's working, if you want to add authentication you need to create a user in the calibre "user accounts" tab, and put the same user/password into lazylibrarian config, then restart the calibre server

mccorkled commented 6 years ago

Thank you. I am glad to have this all sorted. Is there a way we could speak privately about another issue?

Thanks again.

mccorkled commented 6 years ago

I have done these steps but the books are still not being sent to Calibre.

philborman commented 6 years ago

A debug log should shed more light, certainly should be working, I use it all the time.

mccorkled commented 6 years ago

What would be the best way to force the add process so that it shows up in the log. Looking at it now the log just contains post processing information.

philborman commented 6 years ago

The calibre import is part of postprocessing. Are you trying to get lazylibrarian to send existing books to calibre, if so that won't work, we just send our new downloads to calibre. Calibre has it's own import routines for existing books. Use that first, then tell lazylibrarian to do a libraryscan of the calibre directories.

mccorkled commented 6 years ago

LL snatched the book and sent it to SAB. SAB finished the book and placed it in /completed LL processed the book from /completed to /books

Then nothing happened.

lazylibrarian(1).log

philborman commented 6 years ago

The log shows no books found. In each case it says "0 books/mags processed" so nothing will get sent to calibre. Failed to find an ebook or audiobook in [ directory ]

mccorkled commented 6 years ago

That's weird because it is in fact processing the books and placing them in /books. Perhaps we could do a team viewer session.

philborman commented 6 years ago

Not in the log you uploaded... /books is not found in the log file. Sorry, don't have team viewer, my internet connection is too slow.

mccorkled commented 6 years ago

debug.zip

philborman commented 6 years ago

That debug log is empty, just has the header info about your software versions, no actual log lines. The results from my log are below so you can see what we're looking for. You can see calibre gets called and the reply we get, but in my case as the book already exists it won't import it again.

27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:239 : Found 1 file marked "Snatched"
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:240 : Checking 1 file in /media/Popcorn/Download/complete/Books/
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:249 : One Summer was sent to SABNZBD
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:298 : Looking for eBook One Summer in /media/Popcorn/Download/complete/Books/
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:308 : Checking extn on One Summer LL.(17262366)
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:322 : 100% match One Summer : One Summer
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:333 : processDir <type 'unicode'> u'/media/Popcorn/Download/complete/Books/One Summer LL.(17262366)'
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:370 : Found folder (100%) [/media/Popcorn/Download/complete/Books/One Summer LL.(17262366)] for eBook One Summer
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:405 : Found match (100%): /media/Popcorn/Download/complete/Books/One Summer LL.(17262366) for eBook One Summer
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:411 : Processing eBook 17262366
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:processDestination:1020 : Importing One Summer - Bill Bryson into calibre library
27-Oct-2017 13:25:06 - DEBUG   :: POSTPROCESS : postprocess.py:calibredb:941 : [u'calibredb', 'add', '-1', '--with-library', u'/media/Popcorn/Calibre Library/', u'/media/Popcorn/Download/complete/Books/One Summer LL.(17262366)']
27-Oct-2017 13:25:16 - DEBUG   :: POSTPROCESS : postprocess.py:processDestination:1046 : calibredb reports: Backing up metadata
Notifying calibre of the change
27-Oct-2017 13:25:16 - DEBUG   :: POSTPROCESS : postprocess.py:processDestination:1047 : calibredb returns: The following books were not added as they already exist in the database (see --duplicates option):
    One Summer: America 1927:
          /media/Popcorn/Download/complete/Books/One Summer LL.(17262366)/One Summer - Bill Bryson.epub
27-Oct-2017 13:25:16 - ERROR   :: POSTPROCESS : postprocess.py:processDir:565 : Postprocessing for One Summer - Bill Bryson has failed: Calibre failed to import Bill Bryson One Summer, no added bookids
27-Oct-2017 13:25:16 - WARNING :: POSTPROCESS : postprocess.py:processDir:586 : Residual files remain in /media/Popcorn/Download/complete/Books/One Summer LL.(17262366).fail
27-Oct-2017 13:25:16 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:597 : Scanning 1 entries in /media/Popcorn/Download/complete/Books/ for LL.(num)
27-Oct-2017 13:25:16 - DEBUG   :: POSTPROCESS : postprocess.py:processDir:633 : Skipping extn One Summer LL.(17262366).fail
27-Oct-2017 13:25:16 - INFO    :: POSTPROCESS : postprocess.py:processDir:638 : 0 books/mags processed.
philborman commented 6 years ago

From the video it looks as though the various parts are mostly working. I'm not sure, but it looks like lazylibrarian still thinks the books are "snatched". Lazylibrarian knows they were sent to SAB, but hasn't noticed the download has completed. We're not getting the info we need zipped up in the logfile as we think the logfile doesn't exist (log says /config/log/lazylibrarian.log does not exist). I think this must be a docker specific issue. Something to do with the folder names inside docker vs the folder names outside maybe?

I think by default the logs are in a subdirectory of config called Logs (capital L), can you check where your logfiles are and make sure the setting in lazylibrarian config matches? It might be something similar with the download directory, lazylibrarian is looking in the wrong folder?

I don't use docker myself, but I will download a copy and have a play...

philborman commented 6 years ago

Found the missing log issue, fixed in latest version. Hopefully a working debug log will show us what's going on. I think your logs were in /config/Logs but docker was defaulting to /config/log

mccorkled commented 6 years ago

debug.zip

Thanks!

philborman commented 6 years ago

ok, debug log was useful... Seems we are not calling calibre for the books. Two tests need to pass for us to call calibre,

  1. Download is an ebook (not audiobook or magazine)
  2. calibredb is set in the lazylibrarian config.

From the log it looks like there is no calibredb set, as some are definitely ebooks eg Orwell_George-Animal_Farm.epub Golden. Arthur - Memoirs of a Geisha.epub but others appear to have no ebook or audiobook in a valid format. We use the extension to check against the configured extensions in lazylibrarian config, eg these directories... /completed/H G Wells - The Time Machine LL.(2493).1 /completed/P D James - The Children of Men LL.(41913).1 (and many others) seem not to contain a file ending in .epub or .mobi (or whatever is set in your config for valid ebook types) and one of the copies of P D James - The Children of Men LL.(41913) contains an audiobook which we can't process because the audiobook library folder is not set.

Can you check you have calibredb set in config Can you also check some of the folders above where we think there is no ebook. Is there an ebook in there that we are not finding?

mccorkled commented 6 years ago

I believe we are getting closer. I changed the "eBook Library Folder" from /books to /completed I also added "Calibredb import program:" to /books where the Calibre db is stored. In your above example you had for that setting.

debug.zip

philborman commented 6 years ago

There are a few config changes needed, I think.

eBook Library Folder: should be the final destination for the books, and the place calibre uses as its library. It should contain the calibre database (metadata.db) and all the books in author folders.

Download Directory:should be where SAB puts the books it downloaded, this is where lazylibrarian postprocessor looks for completed downloads. The postprocessor moves the books from Download Directory to eBook Library Folder (either by itself, or using calibredb). In recent versions Download Directory: is now Download Directories: so you can have more than one. Might be needed if SAB and Transmission are not on the same machine, for example.

Calibredb import program:should be your calibredb executable. It's part of calibre. On a linux system you can go to a terminal and type calibredb --version to check it's installed and in your path. If it comes back with a version number just put calibredb in the import program box. If calibredb --version comes back with an error you will need to find where calibredb is on your system and put the whole path in the import program box, eg /opt/calibre/bin/calibredb

I might add config info to the debug log as it should help tracking these sorts of issues down.

mccorkled commented 6 years ago

I tried both calibredb and /opt/calibre/bin/calibredb and I am getting this error. I don't think I can map /opt in Docker. Can you check on the version you were testing on?

2017-10-31 09:30:04 | ERROR | POSTPROCESS | postprocess.py | processDir | 568 | Postprocessing for The Oh She Glows Cookbook - Angela Liddon has failed: calibredb rc 1 from /opt/calibre/bin/calibredb

philborman commented 6 years ago

Are you using a docker with calibredb built in, like https://hub.docker.com/r/thraxis/lazylibrarian-calibre/ or do you have a separate version of calibredb that you're talking to? If it's separate you need to tell lazylibrarian where you installed it. The thraxis/lazylibrarian-calibre one is /opt/calibre/bin/calibredb

mccorkled commented 6 years ago

I am using this https://github.com/aptalca/docker-rdp-calibre

philborman commented 6 years ago

That's fine, but which lazylibrarian docker are you using?

mccorkled commented 6 years ago

https://hub.docker.com/r/linuxserver/lazylibrarian/

philborman commented 6 years ago

I would suggest switching your lazyibrarian docker to https://hub.docker.com/r/thraxis/lazylibrarian-calibre/ which has calibredb built in. You should be able to share the calibre database between the thraxis/lazylibrarian-calibre docker and the aptalca/docker-rdp-calibre as the database is kept outside both dockers

mccorkled commented 6 years ago

I just installed the version you recommend and I do not see calibredb. It appears to have all the same files as the LinuxServer.io version

philborman commented 6 years ago

Calibredb should be /opt/calibre/bin/calibredb in the thraxis version

mccorkled commented 6 years ago

2017-10-31 15:19:14 | INFO | Nothing marked as snatched. Stopping postprocessor.

2017-10-31 15:19:14 | INFO | 0 books/mags processed.

2017-10-31 15:19:14 | WARNING | Residual files remain in /completed/Douglas Adams - The Hitchhikers Guide to the Galaxy LL.(386162).fail

2017-10-31 15:19:14 | ERROR | Postprocessing for The Hitchhikers Guide to the Galaxy - Douglas Adams has failed: calibredb rc 1 from /opt/calibre/bin/calibredb

2017-10-31 15:19:06 | INFO | Search for Wanted items complete, found 1 book

2017-10-31 15:19:06 | INFO | Downloading eBook Douglas Adams - The Hitchhikers Guide to the Galaxy LL.(386162) from https://www.usenet-crawler.com

2017-10-31 15:19:06 | INFO | Douglas Adams - The Hitchhikers Guide to the Galaxy LL.(386162) sent to SAB successfully.

2017-10-31 15:19:06 | INFO | Requesting eBook download: 100% https://www.usenet-crawler.com: hitchhikers guide to the galaxy The Douglas Adams

philborman commented 6 years ago

Can you check your lazylibrarian docker docker exec -it lazylibrarian bash This will open a terminal inside the docker ls /opt/calibre/bin should show calibredb is there, or maybe it's the wrong docker?

mccorkled commented 6 years ago

I do not know how to get into the back end of a Docker. Everything I do is in the /config folder. If the /opt/calibre/bin folder is not mapped in the container then LL cannot use it. I cannot map it because it is not in the /config folder.

philborman commented 6 years ago

You can get into the docker container using the commands I put earlier. I think maybe you are still using the Linuxserver.io version. Those commands should show calibredb is installed which would mean you are on the thraxis version.

Latest lazylibrarian has a "test" button in config to check we can talk to calibredb which will show if there is a config problem.

mccorkled commented 6 years ago

I am definitely using the thraxis version that you recommended and it's not working. On top of that, Goodreads sync is no longer working either. I can't believe that it is this difficult to do what seems like everyone is doing.

philborman commented 6 years ago

I just downloaded a new copy of the thraxis docker and tried it again here. All seems to work fine. The lazylibrarian version inside the docker is a couple of weeks out of date, but you can use the lazylibrarian internal update to get to latest version. This will give you a "test calibredb" button to check you are talking to calibredb correctly. The thraxis docker needs /opt/calibre/bin/calibredb in the box in lazylibrarian config. The /opt/calibre folder is already inside the thraxis docker.

There doesn't seem to be an easy way to tell which docker you are actually running, the one with calibre or the one without. They are both using the same kernel/python/libraries so nothing different shows in the log. I think the only way to tell is to log into the docker and look if the /opt/calibre folder is there.

Not sure why your goodreads sync isn't working, again it's working here, and I think nothing has changed in that code for quite some time.

mccorkled commented 6 years ago

I am on a Synology device and so I have to map the /opt folder within the OS. Since that folder is hidden, I am not sure how to do this. Here you can see examples of how the current folders I am using are mapped.

2

philborman commented 6 years ago

You do not need to map the /opt folder, it is already built into the thraxis docker. Do you know how to ssh into your synology? if so you should be able to run docker images this will show which dockers are installed on your system, you may have more than one version of lazylibrarian with different names as you may have both the thraxis and linuxserver ones installed

docker ps -a will show which containers are available. Look for the one where Image is thraxis/lazylibrarian-calibre.

docker exec -it container_name bash This will open a terminal inside the docker, then

ls /opt/calibre/bin should show calibredb is there

mccorkled commented 6 years ago

Thank you for working with me this long. I believe I have it working. I did the test button and it said it found the db. The problem I am having now, which I mentioned a few days back, is that GoodReads sync isn't working properly for me to test grab some books. When I hit sync, only 1 book at a time is grabbed, then I have to hit sync again.

philborman commented 6 years ago

Do you mean the "Goodreads Sync" button on the "Manage" page? This does not grab books, it syncs your lazylibrarian database have/wanted books with your goodreads library shelves.