Question: How to set up calibre content server

[ethanopp]

I have referenced the wiki but am still a little unclear exactly how the integration with calibre works... https://github.com/DobyTang/LazyLibrarian/wiki/Configuration

Current setup - not working that great:

I have 2 dockers, 1 for LL and 1 for Calibre. The Calibre docker runs calibre as well as the content server, and references the same /eBooks dir as LL.

Calibre also has it's auto-add setting (in the calibre settings) set to the same dir, /eBooks. So LL places the file in /eBooks when the download is complete, and then calibre "auto-adds "it back /eBooks, but creates the sub-directories in there (i.e. /eBooks/author/booktitle/*.epub)

In the LL settings, do I still need to use the calibredb program or is the content server credentials enough on its own? Also, is there a way to test the content server connection from LL?

Thanks!

[philborman]

If you are sharing the books between calibre and lazylibrarian, Ideally you want a lazylibrarian docker that includes calibredb, like the Thraxis one linked to on lazylibrarian git page. That way you can avoid using auto-add and actually have the two programs talking to each other and sharing books.

Lazylibrarian will search for the book, get transmission to download it into a "downloads" folder (temporary download location, not /eBooks). Lazylibrarian then uses calibredb to talk to the content server in the other docker, calibre content server imports the book into the calibre library, moves it into /eBooks/Author/BookTitle and then tells lazylibrarian where to find it so lazylibrarian can link to it too.

That's a bit of a potted summary, but basically how it works.
Auto-add is a relic from before lazylibarian could talk to calibredb.

[ethanopp]

Okay great thanks for explaining this... So does that mean in the Calibre container, should I go into calibre and turn off the autoadd settings?

[philborman]

Yes. Lazylibrarian will tell calibre about each book it downloads, no need for auto-add.

[ethanopp]

Okay 1 last question... in the calibre auto-add settings, I have "Automatically convert added files to the current output format"...

Will calibre still do the file conversion if books are being import through calibredb?

[philborman]

Good question - don't know. Try it and let me know :-) If it doesn't you can use a lazylibrarian custom-notifier to do it before passing it to calibre

[ethanopp]

LazyLibrarian can do this conversion??

The only reason I am really using Calibre is for the conversion (and the content server)...

...I did see a user registration on LL now though?

[philborman]

LazyLibrarian uses the calibre "ebook-convert" program, which is included in the Thraxis docker, I think. There is a file called "example-ebook-convert.py" in the lazylibrarian folder, which can be used as a custom notifier. Whenever lazylibrarian notifies you it has downloaded a book it can also notify this program which will make sure there is an epub and a mobi version of all new downloads. It's fairly well commented so you can adapt it to your needs.

The user registration bit in lazylibrarian is so you can have different users with different access rights, different "read" and "to-read" lists etc. If you are the only user of your lazylibrarian just turn off "enable user accounts" to turn it all off, or set the "single user bypass login" switch in config so you still get the ability to share reading lists with calibre (reading lists are per-user)

[ethanopp]

Interesting... Right now I just put in all of my family's 'To-Read' RSS feeds from goodreads...

Does the LL user login also server as a content server for them?

Ultimately the main 2 things I am looking for is auto-conversion (which you mentioned above seems possible) and a medium for family to download once ready...

...Would love to get rid of calibre if possible as its rather cumbersome

[ChrisWeiss]

You might look into a stand-alone COPD server. You still need Calibre to maintain the DB, but you dont need to leave it running all the time as a server. https://blog.slucas.fr/projects/calibre-opds-php-server/

[Code-Slave]

i use https://github.com/janeczku/calibre-web and like it a lot

[ethanopp]

@ChrisWeiss what do I need calibre for?

LL seems to have its own database and now that users can be set up on it, seems like it serves my purposes.... Something that can automatically download books and distribute them to end-users... COPD/calibre-web...really all of calibre just seems unnecessary at this point

[ChrisWeiss]

@ethanopp - You would need Calibre for DB maintenance and (maybe) initial setup. If you're happy with LL's downloading and importing, then maybe you wouldn't need it at all. I have a ton of pre-LL content I'm still managing and I still do a lot of personalization. I'd be curious to know if you are now (or end up) happy with a no-Calibre workflow though, it'd be a good 'end state' for me to shoot for.

[ethanopp]

I'm not sure if that is still accurate - LL create's its own DB and for all intents and purposes should be able to run independently of ever needing calibre to be set up... I did originally start off with calibre prior, but it seems to be way easier to maintain now that its just 1 docker I look over instead of 2...

LL does have a setting to re-name files on library scan refreshes, but I think there is some logic baked in there where if it detects that a folder is 'calibre style' with the "(xxx)" numbers at the end of the dir's it doesn't rename them... But in the end it doesn't really matter because LL can still read the book in those dir's.

@philborman can keep me honest here...

[philborman]

That's right, we don't try to rename anything if it looks like a calibre directory, as it would break calibre. The calibre directory names have a number at the end, which is calibres internal book-id for that book so we can't change or remove it.

The LL database is entirely independant of calibre, you don't need to use calibre with it at all, though there is some limited integratin if you want to "share" one directory structure of books with calibre. Saves having multiple copies of books.

One limitation of using LL as a content server is that the [Open] button only feeds one book, no choice of format, which could be an issue if say you use epub on android and your wife uses mobi on a kindle.

Multople buttons (one for each format) looks messy on a table view, could maybe have a pop-up if there are multiple choices to select epub/mobi or keep each user's preference and always feed them the same type?

[barbequesauce]

Following this conversation with interest.

I'm not sure on the role of Calibre here. I have LL for books with Ubooquity as an OPDS server. Is Calibre a replacement for Ubooq? Either way, what does Calibre bring to the table that LL+Ubooq doesnt have (or vice-versa)?

[ethanopp]

@philborman Maybe admin can set default format for 'open' to download, and users can also set their preference in their own settings?

...I also like the idea of a dropdown, so maybe the button defaults to whatever the preference is set to, and an arrow next to the button can open the dropdown so they could still select the other options if desired

[philborman]

Currently working on it this way... System default is a "preferred" type as already done. Prefer left-most type found in the ebooks type list. If user accounts are enabled, user can specify their preferred type, automatically feed them that type if possible. If that type doesn't exist, or they haven't given a preference, show a popup of available types. System default only gets used if no user accounts or using legacy interface, so we don't break anything.

[ethanopp]

@philborman - is it possible to run the "ebook-convert" program through the main repo?

Would prefer to not have to set up a forked one since I am no longer using calibre :)

[philborman]

The main lazylibrarian repo does not include ebook-convert, that's part of calibre. Depends what you want to do, if it's epub to mobi there is a kindle program from amazon called kindlegen that does the job, but it only does TO mobi not FROM mobi

Not sure how stand-alone the calibre ebook-convert program is, maybe you don't need a full calibre install to use it, just the ebook-convert program?

[ethanopp]

Yeah we're all on kindles so just need to get everything TO .mobi...

Is kindlegen something that has a CLI that can be added into LL or is it all done manually?

Wouldn't be the end of the world if I needed to run Thraxis because its definitely still better than running the entire calibre program in a separate docker, don't need all of calibredb and the other things that are in Thraxis though and trying to stay as minimal as possible.

ebook-convert would be a great add to the main repo if it can be run independently of calibredb... Would think there are other people out there that would find these feature useful as well, as for some older books it can be hard to find in .mobi and with my current setup the only way to ensure .mobi is added is to filter everything else from being downloaded in the first place

[philborman]

kindlegen is here... https://www.amazon.com/gp/feature.html?ie=UTF8&docId=1000765211 Instructions in the README

Can't really parcel it up with lazylibrarian, or ebook-convert either, as both need different files depending on your operating system, not written in python.

[ethanopp]

Okay thanks - In the event I do go with Thraxis... Is there a guide on how to set up the custom notification? ...where is the script?

[philborman]

The script is in the lazylibrarian root folder, called example_ebook_convert.py Just put the full path to it in lazylibrarian config in the custom notifiers script box and set it to notify on download (not notify on snatch) You might need to check the location of the calibre ebook-convert in the thraxis docker as it may not be in the system path, in which case you will need to edit the location at the top of the example script.

[ethanopp]

Ahh...confusing.

The main repo has the script to run but not the program, and the thraxis docker has the program but doesn't come with the script?

Am I missing something here?

[philborman]

Hee hee ! Main repo can't include the program for reasons outlines earlier. Didn't know thraxis does not include the script though... wonder why.

You can download the script from the lazylibrarian homepage by going to code / raw and cut and paste the text

[ethanopp]

Hah no worries just wanted to make sure those were correct steps... I have it in there now but when I click 'test custom' it fails...

Logs show the following: `

2018-04-09 14:35:38	ERROR	WEBSERVER	custom_notify.py	_notify	96	Custom notifier returned 1: res[] err[Not a download event ]
2018-04-09 14:35:38	DEBUG	WEBSERVER	custom_notify.py	_notify	36	Custom Message: Test
2018-04-09 14:35:38	DEBUG	WEBSERVER	custom_notify.py	_notify	35	Custom Event: Test

`

Is that expected since nothing downloaded and I am manually trying to run the script from the test button?

This is on Thraxis and I have: converter = "opt/calibre/ebook-convert" # if not in your "path", put the full pathname here in the script

[philborman]

Yes that looks ok, you are connected to the notifier, and it correctly says that it has nothing to do as you have not downloaded anything. I might add some extra messages to make that clearer.

[ethanopp]

Okay - is there a way to run it on the entire library of already downloaded books?

[philborman]

No, not really. Would not be too hard to write a short script to do it though. You can use ebook-convert directly instead of the notifier, ebook-convert book.epub book.mobi

[philborman]

ok, got a short script here that does what you need - will post it tomorrow with instruction, getting late now and could do with a bit more testing...

[ethanopp]

Just threw this together but its giving me 'No such file or directory'

import sys
import os
import subprocess

converter = "opt/calibre/ebook-convert"  # if not in your "path", put the full pathname here

books_parent_dir = '/library/eBooks' # change to your dir

for root, subFolders, files in os.walk(books_parent_dir):
    for name in files:
        if name[-5:] == '.epub':
            epub = (os.path.join(root, name))
            mobi = epub[:-5] + '.mobi'
            params = [converter, epub, mobi]
            print(params)
            try:
                res = subprocess.check_output(params, stderr=subprocess.STDOUT)
                print("Created mobi for %s" % files)
                #exit(0)
            except Exception as e:
                sys.stderr.write("%s\n" % e)
                #exit(1)

[ethanopp]

Okay great! Thanks

[philborman]

Actually yours is pretty much there. Mine was querying the database to get the list of epubs, so is more complicated but might be slightly faster. Yours just needs a few minor tweaks... The "no such file" error is because you are missing the / at the start of /opt/calibre/ebook-convert I would also add a test to see if the mobi already exists

so...

import sys
import os
import subprocess

converter = "/opt/calibre/ebook-convert"  # if not in your "path", put the full pathname here

books_parent_dir = '/library/eBooks' # change to your dir

for root, subFolders, files in os.walk(books_parent_dir):
    for name in files:
        if name[-5:] == '.epub':
            epub = (os.path.join(root, name))
            mobi = epub[:-5] + '.mobi'
            params = [converter, epub, mobi]
            if not os.path.exists(mobi):
                print(params)
                try:
                    res = subprocess.check_output(params, stderr=subprocess.STDOUT)
                    print("Created mobi for %s" % name)
                    #exit(0)
                except Exception as e:
                    sys.stderr.write("%s\n" % e)
                    #exit(1)

[ethanopp]

Ahh the '/'! Knew I was close...

Good point on the mobi test too - don't need to be converting if epub and mobi already exist.

Looks like its working - thanks for the tips

[philborman]

No probs :-)