search

kiwix / kiwix-tools

Command line Kiwix tools: kiwix-serve, kiwix-manage, ...
https://download.kiwix.org/release/kiwix-tools/
GNU General Public License v3.0
433 stars 85 forks source link

Create Kiwix-serve (API) documentation #498

Closed kelson42 closed 1 year ago

kelson42 commented 2 years ago

In particular for the API Web of kiwix-serve

kelson42 commented 1 year ago

@veloman-yunkan This is now really needed by @rgaudin to update the infrastructure of https://library.kiwix.org. I would use read-the doc to write the documentation properly and keep track of everything in markdown in the kiwix-tools repo. I know that actually the code is in kiwix/libkiwix, but not sure this is a good idea to put all of these there. We could have an end-user doc ffor the kiwix-tools here and focus on dev stuff in the libkiwix repository.

rgaudin commented 1 year ago

Here's something I needed to sort out for my own needs. That may or may not help you. Note that this is far from an endpoint documention as what would be interesting is the list of parameters which I didn't need here. Search one for instance has many.

Path Description Changes with
^/$ home page html template release
^/.+ redirects to ^/content/.+ (matched last) consider always (redirect)
^/catalog/root.xml$ [legacy-v1] Library OPDS catalog library
^/catalog/searchdescription.xml$ [legacy-v1] Library OPDS metadata release
^/catalog/search.+ [legacy-v1] Library OPDS search results library
^/catalog/v2/root.xml$ Library OPDS catalog library
^/catalog/v2/searchdescription.xml$ Library OPDS catalog metadata release
^/catalog/v2/entry/{bookId}$ Library OPDS entry never* or release (should OPDS API change)
^/catalog/v2/entries$ Library OPDS entries list library
^/catalog/v2/partial_entries$ Library OPDS patial entries list library
^/catalog/v2/categories$ Library OPDS categories list library
^/catalog/v2/languages$ Library OPDS languages list library
^/catalog/v2/illustration/{bookId}?size={size} Book Illustration never*
^/catch/external?source= Link to external URL proxy page release
^/content/ ^/content/{bookName}/{entryPath?} direct access to ZIM content. used in viewer never (dated book name). Etag'd requests
^/random?content={bookName} redirect to a random entry in ZIM always
^/raw/{bookName}/[meta\|content]/ Raw metadata or Entry never (dated book name). Etag'd requests
^/search?pattern=.+ library-wide search results (nb books <= searchLimit) library
^/search?content={bookName}&pattern={query} single-ZIM search results never (dated book name) or release (code up)
^/search/searchdescription.xml Search metadata release
^/suggest?content={BookName}&term=&start=&count=$ Entry suggestions for a ZIM never or release (code update)
^/viewer$ viewer.html release
^/viewer/.+$ ^/skin$ ^/skin/.+ vewer/skin resource release (has ?cacheid)
^/viewer_settings.js cli-params based settings for viewer session

Using this to bring attention to side issues:

mgautierfr commented 1 year ago

/search/searchdescription.xml and /catalog/searchdescription.xml are identical. Given it's legacy and we never documented API (and our history or API changes), maybe we can drop the former?

/search and /catalog serve two different purposes:

Both have a searchdescription.xml which describe how to search (which can be use by exemple to add them as a search engine to your browser). They are not the same

/content/{bookName}/ and /raw/{bookName}/content/ currently return exactly the same content. Code doesn't explain why we have two similar endpoints and what's the reasoning behind it? I'd guess flexibility as viewer depends on /content but not sure.

/raw/{bookName}/content/ garanty you to always return you the raw content in the book and nothing more. This may be not the case for /content/{bookName}. For now the content is pretty close but it may change in the future. At least in the past this was not the same (/content/{bookName} replaced absolute url or introduce taskbar). For now this is only one difference: /content/{bookName} handle the main page url (/content/{bookName/) and redirect to the article registered as main page in the same. /raw/{bookName}/content/ is served only if there is an article with "/" path.

rgaudin commented 1 year ago

OK ; thanks for the clarifications

rgaudin commented 1 year ago

btw I realize we can access all metadata of a book either via the OPDS entry and thus get the Library versions (maybe edited) and the /raw/xxx/meta endpoint where we'd get the in-ZIM values.

Only exception being the illustration that is only available via OPDS. I think we missed discussing this in the related https://github.com/kiwix/libkiwix/issues/631

Is that on purpose? Should we return raw illustration as well?

mgautierfr commented 1 year ago

There is no exception for Illustration. But raw do not do any smart thing (as fallback and compatibility), you have to use the exact url. Which url are you using for illustration ?

rgaudin commented 1 year ago

Ah you're right it's working as expected. Not sure how I got it wrong ; maybe fooled by curl not returning binary data on the terminal

kelson42 commented 1 year ago

@veloman-yunkan Unfortunately doc is still empty at https://kiwix-tools.readthedocs.io/en/latest/?badge=latest

veloman-yunkan commented 1 year ago

@veloman-yunkan Unfortunately doc is still empty at https://kiwix-tools.readthedocs.io/en/latest/?badge=latest

@kelson42

@mgautierfr has activated readthedocs for kiwix-tools for the documentation branch. I signed in to readthedocs.org with my account but don't have access to @mgautierfr's readthedocs project so that I can activate the main branch too.

kelson42 commented 1 year ago

@veloman-yunkan What is your email address used for readthedocs? I have access myself, here is what I have: image

The "documentation" version should not exist. It should be put on "latest", and "stable" for releases. All three version are active, but only "documentation" has the right content.

rgaudin commented 1 year ago

There was nothing in the PR about RTD. My understanding was that auto build+publish of the doc to RTD would be a separate PR. What you're seeing is probably a manual upload.

kelson42 commented 1 year ago

@veloman-yunkan This would be

There was nothing in the PR about RTD. My understanding was that auto build+publish of the doc to RTD would be a separate PR. What you're seeing is probably a manual upload.

@rgaudin This would be pretty unfortunate, obviously this has to be integrated to the CI/CD.

veloman-yunkan commented 1 year ago

@veloman-yunkan This would be

There was nothing in the PR about RTD. My understanding was that auto build+publish of the doc to RTD would be a separate PR. What you're seeing is probably a manual upload.

@rgaudin This would be pretty unfortunate, obviously this has to be integrated to the CI/CD.

https://github.com/kiwix/kiwix-tools/pull/586#issuecomment-1361328959:

readthedocs pull and compile the documentation so we don't have any CI to do on our side.

rgaudin commented 1 year ago

readthedocs pull and compile the documentation so we don't have any CI to do on our side.

Ah I remember now reading that comment. Fantastic!

veloman-yunkan commented 1 year ago

@veloman-yunkan What is your email address used for readthedocs? I have access myself, here is what I have: image

The "documentation" version should not exist. It should be put on "latest", and "stable" for releases. All three version are active, but only "documentation" has the right content.

@kelson42 I guess I don't have the right permissions. There are no Edit buttons in the Versions table at https://readthedocs.org/projects/kiwix-tools/. My username at RTD is the same as here.

veloman-yunkan commented 1 year ago

@kelson42 Only you, @mgautierfr and @rgaudin are listed as Maintainers at https://readthedocs.org/projects/kiwix-tools/

rgaudin commented 1 year ago

Was due to master vs main @kelson42 that's on you 😉 Fixing it and adding @veloman-yunkan

rgaudin commented 1 year ago

https://kiwix-tools.readthedocs.io/en/latest/?badge=latest 👍