Any plans to support OPDS feeds with credentials?

[johnruck]

My OPDS server uses credential login to attempt to secure the server. Is there any chance you will support OPDS servers with login requirements?

[johnruck]

Or is there some form of including the login credentials in the URI that you support?

[isaaclyman]

@johnruck I'd like to add credentialed requests to the app, I just haven't been able to find a feed to test against. It would be a snap to build, most likely. What OPDS server do you use? How do the credentials work?

[johnruck]

I am using a calibre server (it is hiding behind a nginx proxy manager to get https on it) so it is fairly standard - I think it just does the standard html password request handshake. The password login is done on the calibre server setup side. Docs for the calibre server are here - https://manual.calibre-ebook.com/server.html . Interestingly on that webpage it mentions that it takes a http://username:password@url syntax which I tested via a browser that worked, but inside bookoscope it came back with a 400 error code (instead of the 401 code without the username:password@ syntax)

[isaaclyman]

@johnruck Are you able to provide a copy of the authentication document for your calibre server, as described here? Basically I'm just trying to find out if calibre supports Basic authentication or only OAuth.

[johnruck]

My reading of the calibre docs is that it is basic authentication. I have had to spend a few days resurrecting one of my servers which is why I have been delayed in replying. Let me see if I can setup a dummy calibre setup with authentication to let you work against. (Dummy setup because it will just be the default install with the users manual in the system, but I will set it up for authentication so you can get your code working against it)

[isaaclyman]

Thanks for your help. I'm working on implementing basic auth today.

[isaaclyman]

@johnruck I've got something that should work. Can I add you to the beta program for the app so you can get the changes right away and see if they'll work for you? I would just need to know whether you're on iOS or Android and what your account email is. If you don't want to share it here, you can email me at bookoscope (at) isaaclyman.com.

[johnruck]

I pulled down the updated app from the iOS app store - I am still getting error 400 with either configuration (that is, I have one configuration where I have filled in the username and password and a different one with the username/password encoded into the url - both give an error 400).

[isaaclyman]

It may be the way the URL is formatted. Can you copy an example URL here?

[johnruck]

tomorrow I will setup the test server for you, it might also be something with my password (it is very complex and long) so once I know it fails with a simpler password I will get back with you.

[johnruck]

ok, a little debugging on my end and some small progress. It ends up that in calibre advanced settings you can specify the authentication to be used - it defaults to "auto" which appears to be digest instead of basic. Forcing basic allows your app to connect (so that issue is identified now). The odd thing is your app is reporting no books, on either my test setup (with its default users guide loaded) or my real setup (with over 2700 books in it at the moment) so I guess there is something in the calibre opds that you are not parsing properly. For now I will leave my test opds server up for you to test against.

url - https://opdstest.silverblazegaming.com

username = caltest password = Alpha6871

there is 1 one ebook in there - Quick Start Guide, which is described on each of the pages off of the initial opds entry page.

Thanks again on this.

[isaaclyman]

Thanks for setting that up. I've pulled a couple of documents from that test server to use as unit tests.

As a progress update, it looks like calibre doesn't generate rel="self" links in the OPDS documents, which makes it tricky to determine how one location is related to the next. It seems like no ebook server fully implements the OPDS standard, but all of them in different ways. I'll let you know when I have a fix.

[isaaclyman]

Okay, I've submitted a fix. The latest version of the app is able to discover and parse all the feeds on that server (the root being https://opdstest.silverblazegaming.com/opds - note the path), with the exception of a couple that appear to be bad links on calibre's side. It finds and caches the Quick Start Guide, which is then downloadable through the app. I can't guarantee it will work the same on your other server, but the fundamental issue should be resolved. Let me know how it works for you - if Apple is on top of their reviews like they were yesterday, you should be able to download the update shortly; otherwise, as before, I'd be happy to add you to TestFlight.

Thanks for all your help tracking this down. If you have a ko-fi or a Paypal or something let me know - the least I can do is repay you the cost of the app.

[johnruck]

Yep that worked for me - scary thing was your statistics while doing the crawl - my 2800 book setup required 5555 pages and 5538 pages crawled.

So thanks for getting this working.

While I have this setup for you for debugging, do you want me to switch it over to use digest authentication so that you can figure that out as well (as it seems that is the default authentication out of the box for calibre)?

[johnruck]

one thing I just discovered - attempting to download a book from the server comes back with an authentication request page. Any chance of you detecting that and filling in the username/password you have?

I will say since you are doing the crawl the search functionality is MUCH better than I have seen in any ebook reader with OPDS support.

[isaaclyman]

scary thing was your statistics while doing the crawl - my 2800 book setup required 5555 pages and 5538 pages crawled.

That's pretty standard for OPDS - each book can appear in several different documents, organized by author, genre, title, etc. It's an inefficient format for machine consumption, but it's what we've got.

I will say since you are doing the crawl the search functionality is MUCH better than I have seen in any ebook reader with OPDS support.

Thank you, glad to hear it - that's the reason I built the app.

Any chance of you detecting that and filling in the username/password you have?

There's a limited way to provide headers in a webview, but it looks like platform support is spotty and it may not work most of the time. I've added some best-effort code for the next version release but YMMV.

do you want me to switch it over to use digest authentication

Sure, if you'd be willing. Digest auth is new to me so it may take longer to figure that one out.

[johnruck]

ok, I have switched it over to digest (tell me if you need me to go back to basic if you need to debug something else on it).

And dont worry about compensating me for this - I found your app on the app store because I was wondering if there were any dedicated OPDS apps before I would break down and write my own, so the your app is well worth the price for me.

I had been using KyBook 3 for pulling files via opds before migrating them into the ebook readers I wanted the documents in, so getting your app working where I can remove KyBook 3 (which seems to have become abandonware at this point - at least the backing sync server has had its certificates voided so I dont trust that functionality now).

Thanks again for your app!

[isaaclyman]

The same exact thing happened to me (with KyBook and a couple other apps), but six months ago. :)

I seem to have a working solution for digest auth, but I'm worried I may have broken basic auth in the process. Do you mind switching it back for a moment?

[johnruck]

sorry, stepped a way for a minute for lunch and break . I have switched it back to basic now.

[isaaclyman]

No problem. Everything checks out, I'll ship the update and close the issue. Thanks for all your help.

[johnruck]

no problem and once again thanks for making this app in the first place :)

[isaaclyman]

You're welcome! Let me know if you run into any more trouble or have any suggestions.

[johnruck]

btw - I am using MapleRead SE for epub types of books and GoodReader Pro for PDFs (so that I can annotate the PDFs) these days as both of those will sync bookmarks and such via icloud as I read my books on lots of different devices. I dont like that you cant really organize the books in MapleRead but it is the best one I have found that has some sort of still working sync functionality.

[isaaclyman]

Thanks for the tip, I'll check those out. My latest conspiracy theory is that all the PDF readers on iOS have been bought out by the same company and are gradually becoming reskinned versions of each other at near identical price points. I've been reduced to reading all my RPG books on the Kobo app. I'd much rather have a simple, dedicated PDF reader I can pay for once.

[johnruck]

I use Kobo as well for the content that I have from them, but it doesnt really like non official content from them (doesnt sync it from what I could see) - I tried that before I found MapleRead SE. I am using GoodReader for my RPG books as well as all of my computing ebooks so that I can annotate them in place and thus keep my notes in the file itself. (For me Calibre is my "master" catalog and warehouse, with files going from calibre into dedicated reader spaces (like Komga for comics/manga to be read with the ios Panels app as that supports streaming pages at a time for cbz/cbr files). Currently in the process of rebuilding my Calibre library with my new system, which is why the book count is constantly growing right now (just cleared 2900 books today). This way the "pristine" originals are in calibre with all of my notes in the versions of the files elsewhere.

[johnruck]

my only annoyance at the moment is the pdf reader, GoodReader Pro is only on iOS, not windows so I havent come up with what I am going to use to sync/annotate on my laptop. I tried Adobe Digital Editions but it wants to absorb each pdf you read into its own storage area which breaks the access on the iOS side. So still looking for that app......

[johnruck]

did you ever push the version that was attempting to do the authentication for downloads? I couldnt tell at the time if that was pushed or not. If you did it isnt working,

[isaaclyman]

Yes, that version is pushed. It doesn't work for me either, but that's a platform issue. There just isn't a reliable way to send headers to a webview as far as I can tell.