search

DobyTang / LazyLibrarian

This project isn't finished yet. Goal is to create a SickBeard, CouchPotato, Headphones-like application for ebooks. Headphones is used as a base, so there are still a lot of references to it.
729 stars 72 forks source link

OPDS enhancements #1644

Closed Code-Slave closed 5 years ago

Code-Slave commented 5 years ago

This is through Moon+ Is there a way to style the rss? When i browse calibre opds is is styles and colored. with top nav etc

philborman commented 5 years ago

Nothing in opds spec about styling and no other files are sent, just plain xml data. I guess moon+ does something special when it detects the opds is from calibre? Can you post the opds xml from calibre so we can see if we can fake ours to look similar?.

I just added Ratings as a nav item, should be able to add read/to-read fairly simply

Code-Slave commented 5 years ago

Im still trying to find a ebook reader for linux or osx that browses opds. I cant pull the xml from my android tab.

barbequesauce commented 5 years ago

Just use curl...

curl http://lazy.lib.server/opds > opdsfeed.txt

Code-Slave commented 5 years ago

yea, what im trying to see is if calibre or the ereader renders different. as the display for calibre opds is very different from ll opds. When i go to the ops server in a browser its standard xml for each

barbequesauce commented 5 years ago

Yeah sorry, the phone cut off half my response.

curl http://lazy.lib.server/opds > opdsfeed.txt.LL
curl http://calibre.server.opds > opdsfeed.txt.calibre

Of course, confirm the calibre feed address. What Phil is heading to is that the OPDS feed is not quite as standardized as you might expect; hopefully we will see "things" in the Calibre feed that are missing in the LL feed to account for the rendering difference.

philborman commented 5 years ago

Could be calibre is sending different/more info than us, or could be the reader recognises it as a calibre feed and handles it differently, applies different rendering?

philborman commented 5 years ago

Read and To-Read flags are in the next release, you need to add your lazylibrarian userid to the opds feed so we know who you are, eg http://lazy.lib.server/opds/ToReadBooks?user=12345 Your userid is on the user profile page

Code-Slave commented 5 years ago

Yea i get that. what i cant show is how the diff apps render. I think moonreader does calibre specific stuff

philborman commented 5 years ago

If it does it probably looks at the feed name to know it's calibre? As long as we are sending compatible info maybe we can fake it?

barbequesauce commented 5 years ago

That's why I was thinking to see both feeds side-by-side....

WillowMist commented 5 years ago

Wait, are we talking about COPS? (https://github.com/seblucas/cops) -- That's the Calibre OPDS I'm familiar with. If you browse there, that's not actual OPDS output. The COPS server gives you a web interface PLUS an OPDS feed. The actual opds feed on a COPS server is /feed.php

I'm not sure it's realistic to try to incorporate an actual web interface into the OPDS reader. My ultimate, someday, is to do a web interface that will work with Mylar and LL's opds feeds, and possibly include a reader, but that's no farther than the vision stage, at this point.

Code-Slave commented 5 years ago

@DarkSir23 No the actual feeds and how moonreader etc render them

WillowMist commented 5 years ago

Can you post screenshots, perhaps? But also please clarify for me which OPDS you're using.

yabdali commented 5 years ago

The latest version showing errors on various eReader apps using OPDS.Tested on Moon+ Reader, Aldiko, FBReader, PocketBook. Viewing recent ebooks and audiobooks in Moon+ is ok but if I click the pagination button I get ann error. Other apps seem to do pagination as I scroll down, initial list of ebooks and audiobooks load normal but I get a flashing error message as I scroll further.

RecentAudiobooks behavior using different apps: -FBReader: clicking a recent audio book details displays info but no dowload option. -PocketBook: clicking on Recent AudioBooks doesn't show show the list of books, empty page. -Aldiko: clicking a recent audio book details displays info and dowload option.Clicking download shows its donloading with progress bar. -Moon+: Clicking an audiobook in the list of recent audiobooks shows a prompt for download.

Best Rated Books on Moon+, FBReader, Aldiko show internal 500 error page. PocketBook displays open browser popup taking me to /opds?cmd=RatedBooks with the error below displayed. Would you be able to use https://github.com/Feedbooks/opds-test-catalog for testing OPDS compliance and troubleshooting potential issues?

The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last): File "/volume1/web/lazylibrarian/cherrypy/_cprequest.py", line 670, in respond response.body = self.handler() File "/volume1/web/lazylibrarian/cherrypy/lib/encoding.py", line 217, in call self.body = self.oldhandler(*args, *kwargs) File "/volume1/web/lazylibrarian/cherrypy/_cpdispatch.py", line 61, in call return self.callable(self.args, self.kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/webServe.py", line 4739, in opds data = op.fetchData() File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 106, in fetchData _ = methodToCall(self.kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 759, in _RatedBooks return self._Books(sorder, **kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 842, in _Books author = makeUnicode(author['AuthorName']) TypeError: list indices must be integers, not str Powered by CherryPy 3.6.0

philborman commented 5 years ago

It's all this same bug I think, error introduced a few days ago. That bug is fixed upstream, but not pushed to master branch yet, should be tomorrow.

yabdali commented 5 years ago

Thanks for the heads up, will test it once available. Not sure if you would consider to use https://github.com/Feedbooks/opds-test-catalog for testing OPDS compliance and troubleshooting potential issues? This can make it easier to assess any issues...

philborman commented 5 years ago

That looks like a catalog to test clients, not servers. I ran our output through https://opds-validator.appspot.com/

philborman commented 5 years ago

Not sure what those ebook readers are supposed to do if they get an audiobook. I guess fbreader is correct in that it shows a list with no download links (no formats it can handle). Pocketbook is maybe more strict, doesn't show books at all if they don't have the right format, and moon+ and aldiko will let you download even though they can't handle it?

Aldiko pagination is broken, that's a long standing aldiko bug that's never been fixed.

philborman commented 5 years ago

Just added a work-round for aldiko pagination issues. Just the "no images" I can't figure out yet. Aldiko delegates image loading to Android without handing over authentication data so if we have opds auth turned on we refuse to send the images

yabdali commented 5 years ago

I have been getting some erors. Below is what I get by directly accessing OPDS using the web browser.

/opds?cmd=RatedBooks 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last): File "/volume1/web/lazylibrarian/cherrypy/_cprequest.py", line 670, in respond response.body = self.handler() File "/volume1/web/lazylibrarian/cherrypy/lib/encoding.py", line 217, in call self.body = self.oldhandler(*args, *kwargs) File "/volume1/web/lazylibrarian/cherrypy/_cpdispatch.py", line 61, in call return self.callable(self.args, self.kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/webServe.py", line 4745, in opds data = op.fetchData() File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 121, in fetchData _ = methodToCall(self.kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 882, in _RatedBooks return self._Books(sorder, **kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 971, in _Books author = makeUnicode(author['AuthorName']) TypeError: list indices must be integers, not str Powered by CherryPy 3.6.0

/opds?cmd=RatedAudio 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last): File "/volume1/web/lazylibrarian/cherrypy/_cprequest.py", line 670, in respond response.body = self.handler() File "/volume1/web/lazylibrarian/cherrypy/lib/encoding.py", line 217, in call self.body = self.oldhandler(*args, *kwargs) File "/volume1/web/lazylibrarian/cherrypy/_cpdispatch.py", line 61, in call return self.callable(self.args, self.kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/webServe.py", line 4745, in opds data = op.fetchData() File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 121, in fetchData _ = methodToCall(self.kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 1007, in _RatedAudio return self._Audio(sorder, **kwargs) File "/volume1/web/lazylibrarian/lazylibrarian/opds.py", line 1064, in _Audio author = makeUnicode(author['AuthorName']) TypeError: list indices must be integers, not str Powered by CherryPy 3.6.0

philborman commented 5 years ago

Seems to work fine here. We should be able to address author[ ] by name or position so author['AuthorName'] should be the same as author[0] I'm guessing a difference in python or sqlite3 versions? Can you post the output from lazylibrarian config "system info" button If that's what it is there will be other places in the lazylibrarian code with the same problem. Also if you are up to a bit of editing we could try changing the opds.py file to use positions, I could guide you through it, just to confirm that's where the problem is?

philborman commented 5 years ago

ok, managed to reproduce it. You have a book with an authorid that's not recognised in the database. Not sure how yet, shouldn't happen. Looking into it. Will add some code to remove orphaned books at startup to correct it until I can figure out how it happens.

WillowMist commented 5 years ago

regarding no-image... Would it be possible to create a second location for cached images without authentication, and use it in the opds feed rather than the normal cached image location? Not two sets of images, just two references in the WebStart.py?

yabdali commented 5 years ago

Seems to work fine here. We should be able to address author[ ] by name or position so author['AuthorName'] should be the same as author[0] I'm guessing a difference in python or sqlite3 versions? Can you post the output from lazylibrarian config "system info" button If that's what it is there will be other places in the lazylibrarian code with the same problem. Also if you are up to a bit of editing we could try changing the opds.py file to use positions, I could guide you through it, just to confirm that's where the problem is?

Here's the info output Startup cmd: ['/bin/python', '/volume1/web/lazylibrarian/LazyLibrarian.py'] Interface: bookstrap Loglevel: 3 Sys_Encoding: UTF-8 git_repo: lazylibrarian git_user: dobytang git_branch: master latest_version: 90060b5f1789cbada42b18ccbd240673e411836b git_updated: Sun Nov 4 21:15:11 2018 current_version: 90060b5f1789cbada42b18ccbd240673e411836b git_host: github.com commits_behind: 0 install_type: source auto_update: 1 Python version: ['2.7.12 (default, Sep 7 2018, 13:17:15) ', '[GCC 4.9.3 20150311 (prerelease)]'] Distribution: ('', '', '') System: Linux Machine: armv7l Platform: Linux-3.10.105-armv7l-with-glibc2.4 uname: ('Linux', 'DiskStation', '3.10.105', '#23824 SMP Fri Oct 12 18:27:43 CST 2018', 'armv7l', '') version: #23824 SMP Fri Oct 12 18:27:43 CST 2018 mac_ver: ('', ('', '', ''), '') urllib3: missing requests: 2.12.4 tls: TLS 1.2 cherrypy: 3.6.0 sqlite3: 3.10.2 unrar: missing: can not find library unrar openssl: OpenSSL 1.0.2o-fips 27 Mar 2018 pyOpenSSL: module missing magic: missing

philborman commented 5 years ago

Tried making the images no-auth, but it seems if the connection needs auth it ALL needs auth, not just some parts of it. Can't find a way around it at the moment, and aldiko have known about the issue for a few years (first reported from calibre users) and no sign of a fix from them, though aldiko 4 i due out soon so who knows...

philborman commented 5 years ago

@yabdali Thanks, found the issue. It's to do with "foreign key" constraints not remaining on. This is used so when you delete an author all their books should be deleted too. Fixed in testing here, also added code to clean up the "orphans" at startup. New version should be out later today.

yabdali commented 5 years ago

@yabdali Thanks, found the issue. It's to do with "foreign key" constraints not remaining on. This is used so when you delete an author all their books should be deleted too. Fixed in testing here, also added code to clean up the "orphans" at startup. New version should be out later today.

Thanks for your continuous support. I run some SQL queries for the books and authors then exported them to CSV and used MS Access to find those orphans. Used the old school approach via Excel to call the API by concatenating the URL with the authorID. All those missing authors were added and now the /opds?cmd=RatedBooks provides XML output without issues :)

I can see that you added the rating as part of the title, I presume you have set it to minimum 4.0? Can you provide an option where we can enter into the OPDS url parameter a value for minimum rating? It can be that if not set then the default is 4.0.

philborman commented 5 years ago

Yes, it's quite simple to do in sqlite, I'm learning all the time. Powerful stuff sqlite... select authorid from books EXCEPT SELECT authorid from authors will list all books that have no author in the author table. Similar code in lazylibrarian now also checks series members with no series, magazine issues with no parent magazine, etc.

yabdali commented 5 years ago

Thanks a lot, its working flawlessly