Comparison Analysis: Calibre-Web vs. Kiwix

zion-fung commented 4 years ago

Team

Project Gutenberg in a Box - Senior Design at Stevens Institute of Technology (2019-2020)

Members

Zion Fung, Vincent Lee, Alex Schlumpf, Sean Trinh, Hariharan Vijayachandran

Primary Client

Eric Hellman, Free Ebook Foundation

Products

Calibre-Web, Kiwix

Purpose

To analyze the advantages, disadvantages, and opportunities of using calibre-web or kiwix in providing robust, efficient, and speedy search capabilities on the Project Gutenberg library on “Internet in a Box”.

Outline and Methodology

This document shall outline a preliminary analysis and first explore the advantages, disadvantages, and opportunities of using calibre-web to accomplish the team’s project goals. The document shall then do the same for kiwix. The preliminary analysis will be derived from the team’s own research and experimentation with each product as well as feedback from the primary client. The goal is to then come up with a preliminary recommendation and product choice. This document will then be distributed to other field experts and applicable parties and be amended with their feedback and recommendations. A final product decision will then be made, after discussion with the primary client and team.

Calibre-Web

Advantages
- Calibre-Web ships with excellent recommendation lists, such as “Most Popular Books” and “Best Rated Books”. Users can also already see the Recently Added Books, Categories of books, and authors and their books.
- Calibre-Web currently allows users to see a list of books they have marked as read and books they have not read and download books.
- Calibre-Web current allows users to create lists or “shelves” of books, serving a need of the project’s primary client.
- Calibre-Web has an advanced search option, allowing users to search based on the book title, author, publisher, publication date, tags (genres, etc.), rating, and description.
- Calibre-Web is open-source.
Disadvantages
- Calibre-Web is supported by one individual developer, making technical requests possibly more time consuming and infeasible. In addition, no contact or current working relationship has been established with the developer of Calibre-Web.
- The migration of the Project Gutenberg library to Calibre-Web is a potential problem as downloading the books from the Project Gutenberg zim on Kiwix and uploading them to Calibre-Web results in a loss of metadata, like the book cover, author, etc.
Opportunities
- Pending a solution to the problem of migrating all of the Project Gutenberg library to Calibre-Web, a potentially easy problem requiring a comparably smaller amount of time to solve, using Calibre-Web would allow the team to focus and work on other aspects of the project, such as the technical aspects and logistics of distributing Internet in a Box with the Project Gutenberg library to the end users.

Kiwix

Advantages
- Kiwix has an extensive open-source software architecture and a large team and network of developers supporting it. The code to scrape book metadata and create ZIMs is also open source.
- A professional working relationship and contact has been made with the developers, maintainers, and owner of the Kiwix software.
- Kiwix currently supports a very fast search on author.
- Kiwix currently allows users to sort books based on their popularity (most downloaded) or alphabetically by title.
- The Project Gutenberg ZIM on Kiwix contains a plethora of useful metadata to support the implementation of more advanced search options.
- Kiwix allows users to read books in HTML format, without opening up a specialized reader or new tab. Kiwix allows users to download books.
Disadvantages
- Kiwix only currently supports a fast search on author. Title can be searched on in the top search bar, but if the user does not know exactly what book they are looking for, results can take a long time to display.
- Kiwix does not currently have advanced search capabilities, something the team would have to implement to accomplish the project goals. While a simple blind search can be implemented, it wouldn’t be as efficient as an indexed search, but indexed searches would require the support of a custom backend.
- Kiwix itself does not come with the ability to create lists of books, a need of the project’s primary client. (It is worth noting that this can be served by Calibre-Web, although teachers would have to download desired books from Kiwix and then upload them to Calibre-Web while losing metadata)
Opportunities
- Using Kiwix would give the team the opportunity to contribute to an open-source project, while implementing sophisticated search solutions using existing technologies such as FlexSearch/ElasticSearch.
- The team would also have the opportunity to continue working with the developers of Kiwix, while gaining valuable real-world programming and software development experience.

Preliminary Product Recommendation

Conclusion
- All things considered, there are clear and obvious pros and cons to using either product. Our initial and preliminary recommendation is to implement search features using Kiwix. Given our current working relationship with the developers and supporters of Kiwix, it would be beneficial for us when we run into issues and roadblocks. Although robust search features are already on calibre-web, we find that the Kiwix architecture is suitable enough for us to easily implement these features on top while surely maintaining useful and important metadata. We also find that the opportunity to work on a substantial open-source project invaluable.
Plan of action
- Work on the “backend” features, such as indexing the metadata and implementing PouchDB.
- Develop more features for the Kiwix front end, such as title search and tabs.
- Our group will iterate, adding more front end features with the support of the PouchDB infrastructure we build.

kelson42 commented 4 years ago

@zion-fung Thank you. I agree with the analysis even if I don't really understand the sentence around ElasticSearch. One thing I can add, you don't have talked at all about Kiwix/openZIM build-in fulltext search. Not sure this is because this is not part of the features you need, or because of lack of knowledge.

What is important to us for now: setup a roadmap with milestones, tasks and deliverables. It is really important the overall project to be split in smaller tasks. One time we all agree on the overall plan, the tasks could be added in this tracker and you could start to make the PRs. By doing so, the together work should be really easier (in particular the PR reviewing process).

Concerning the reading list, if the work on this scraper is a success, I'm committed to implement the reading list on Kiwix Hotspot side before the summer (So @eshellman get what he needs at the end). See https://framagit.org/ideascube/kiwix-hotspot/issues/337

kelson42 commented 4 years ago

@zion-fung @eshellman We don't have heard anything new since 2 weeks. Does a technical roadmap with milestones has been setup? I also guess we can close this ticket?

seantrinh commented 4 years ago

Hi! Sorry for the delay. We have been caught up with the holiday season and finals, projects, etc. We have a rough, high-level completion schedule set up to track our progress.

openzim / gutenberg