search

ajslater / codex

Codex is a web based comic archive browser and reader
GNU General Public License v3.0
202 stars 6 forks source link

OPDS bugs and observations #243

Closed beville closed 1 year ago

beville commented 1 year ago

Intro

I've not had a lot of luck with getting OPDS with Codex to work well, and I realized recently that part of it probably has to do with the large library (over 17K files including over over 2700 series and 96 publishers). To see what might be going wrong for me I created a smaller library with just around 40 files, 3 series and 2 publishers, and set up another instance of Codex on a different port so I can look at both of them at the same time.

I also just set up Komga (with the same small library) as control to see how that presents in OPDS. For clients, I'm mostly using Chunky and Panels on an iPad as the reader apps, there are the only really viable streaming comic readers, but also a few others including Foliate and Thorium on Linux (can't read comics but can browse OPDS), and Yomu on iPad to test out the browsing and navigation.

Once I started playing with Codex OPDS with my small library on Chunky, I began to understand what was confusing me and realized what bugs I'm also seeing with the browsing that I suspect are caused by the large library.

First my notes on what I finally gleaned, and then I'll get into the specific bugs.

Browsing Paradigm Problem

Playing with Codex with the smaller library (without the bugs listed below) made it clearer that Codex's way of representing the library was different the way the most other (usually book) OPDS servers work, and the way that the clients were working. For example with Komga, the initial menu is just a list of virtual folders ("Latest books", "All publishers", "On Deck") and as you click on an item, you're drilling down into a virtual tree where nothing in the branch repeats and all tree nodes terminate in lists of comic issues or empty.

Codex, though, in addition to some new stuff, repeats the same nodes (some of which I was originally perceiving as virtual folder) inside every node. What I didn't realized was that some of them aren't folders, but are toggles. Namely the Series/Folder/Publisher views. I could keep "drilling down" to infinity by tapping those three in sequence. Once I had a smaller library I understood that I wasn't really drilling down, but but was toggling views. Part of the problem is that the apps are just treating this like navigating folders, so it doesn't present clearly to the user (or at least it doesn't to me.)

Some OPDS apps (Yomu on iPad and Thorium) can't even browse Codex at all but work with other OPDS servers.

This is just my two cents, but I'd consider dropping the complex facet-y node things, and go for a more simple representation of virtual folders that don't repeat, since I think that better matches the simple paradigm of the existing clients.

I'd recommend a more basic idea that gets rid of the view/order toggles, and just does simple "folders" with no repeating special nodes:

Newest Issues
    <Issues> (limited number)
Oldest Unread
    <Issues> (limited number)
Publishers
    Series
        <Issues>
Series
    <Issues>
Folders
    <remapping actual folder structure>
        <Issues>

Large Library Bugs

1. Newest Issues can't handle large library

The first problem I hit with my large library was with the "Newest Issues" list: It would never load on any of the various OPDS browsers that I would try. With my tiny library, I do see it load, which leads me to wonder if it's trying to load all 17K files at once, but just ordered by start date? This could be a very simple fix to limit that to just that most recent 50 or 100 issues.

Probably the same for oldest unread but I haven't tried that one.

2. Series View Pagination error?

I see this in both Chunky and Panels. This is hard to describe, but when I start with Codex OPDS, I have the Publisher list (96 items) along the the special nodes. I then click on the "Series View". I get a list of the first 100 (exactly) series, and then all of the publishers again. And some of the special nodes are repeated. I'm unable to get to any more of the series. In Foliate, I see the 100 series, and with a right arrow button at the bottom which implies some pagination, but when I click it, I'm back into the Publisher View again.

3. Folder View Spillover?

This is similar to the above, kinda. My large library doesn't have a super clean structure, and a lot of folders are loosely grouped by month added. And one of the top-level folders is just the most recent stuff until it gets organized (if ever). Call that last one "new_stuff". So when I enter the folder view, I'd expect to see just new_stuff and one other folder, but in addition to that, there's a whole bunch of files from new_stuff at that level. When I dive into new_stuff, I think I'm just seeing the files that weren't shown "above". As I drill into other branches of the mirror folder tree, I see other similar anomalies.

My small test library doesn't show any of these problems.

Thanks again for taking the time to read this! Let me know if I can test anything, or give screenshots or logs or whatnot.

beville commented 1 year ago

Regarding # 3 above, I just realized that I see the same behavior in the web UI too. The actual new_stuff folder has 73 files in it. On the web UI when I change to "Folder View" I see the two top level folders shown (as expected), as well as 45 of the files that should be in new_stuff . Drilling down in new_stuff I see the remaining 28 files. I guess I've seen this for a while, but never really understood what I was seeing.

I'll try to re-create this in a simpler scenario.

UPDATE: I just tried moving all files out of the scanning range of Codex, and saw the folder view empty of those files. I moved them back, and now it looks as it should, will all 73 comics in the right place, with no "spillover". So I kinda repaired things somehow...

beville commented 1 year ago

OK, I recreated item # 3 above. It seems to happen when there are two libraries (root folders) in for Codex. I ran a tagger on the files in one library, so a bunch of files were modified. Maybe since this is happening in a folder that Codex is actively watching, it get confused and sticks them in the no-mans-land of the Folders View "root". I went to the admin panel and did a "Force Update" for that library, and it resolved that problem.

This is definitely not a OPDS problem, and maybe not a real problem. At least not a big problem.

ajslater commented 1 year ago

Bugs

I think these bugs 1 & 2 you've documented have the same root cause and it really breaks OPDS on Codex. Thank you for reporting them. I'll be addressing them in an upcoming release.

Browsing Paradigm Problem

The UI issues you have with OPDS's filters and facets pushed down into regular links is, yeah... i mean, it's confusing. There is an OPDS spec. it has facets and filters. Codex's major idiom is kind of acting like a database view with facets and filters. OPDS reader software in my experience, pays very little mind to the OPDS spec. I would guess that most of them target a single feed idiosyncratic feed source that has also doesn't care much about the OPDS spec. So they all have quirks. And some of them implement facets and filters and some do not. So for those that I can test that do have facets, I use the actual OPDS facet spec an the reader puts them in a special facet UI, but for everything else I create special feed links. To disambiguate these special feed links I've given them special prefix icons, but I'm sure that's not quite enough to avoid confusion as you've well noted. Idk what i'll be doing on that front, but your detailed feedback is appreciated.

beville commented 1 year ago

Yeah, it really still feels like the Wild West when it comes to the readers and feeds. I’d forgotten that you’d added some special cases for some readers.

Realistically, the only iOS comics reader under active development that handles OPDS feeds is Panels. (I’m still mostly using Chunky and my own extremely long-in-the-tooth ComicStreamer experiment, but I’m afraid Chunky is just one iOS update away from breaking) I’m probably going to make some features requests for Panels that I’ll make a point of mentioning Codex in, so maybe they will start doing fixes and development with an eye in this direction too.

Thanks!

ajslater commented 1 year ago

v1.2.4

  1. The Newest query timeing out was a consequence of me overloading the OPDS categories tag with far far too many many to many queries. I trimmed it down to just creators and now it works.
  2. The pagination was just broken. It should now.

I found out that when Chunky encounters pagination it tries to traverse the entire pagination list before displaying anything.

I've left the special facet and sorting entries in the feed for now, but they only display on page 1 so that should get rid of duplicates in Chunky and maybe others readers.