Google Books alternative - Full text browsing and search

ocdtrekkie commented 8 years ago

Project description

I have a lot of ebooks, mostly in PDF format, but some EPUBs as well, and I feel the information in them is often better than on the Internet. But I've got no way to find that information easily. I'd like a self-hosted (ideally web-based) platform that I can just upload all my books, and then search and browse them from wherever I am.

I asked for something like this on HN, and other than general full text search apps to run on my desktop, there really wasn't anything to do this.

Relevant Technology

Needs to support hosting on a Linux box. Should ideally be usable from Windows, Linux, Android, Mac, etc. through a web interface.

Who is this for

Probably at least somewhat savvy users since we're talking about self-hosting here, but hopefully one could admin a server for their family to use or something.

FutureProg commented 8 years ago

@mysticmode Nope, haven't received one yet. I think you can post a join link?

FredrikAugust commented 8 years ago

mail.fredrikaugust@gmail.com :)

ghost commented 8 years ago

@FutureProg I revoked and sent it again. Please check now.

FutureProg commented 8 years ago

@mysticmode I'm not certain what's going on but I still haven't received an invite (and it says my email isn't in the system when I do a password reset). Are you sure you're sending it to nickmorrison09@gmail.com ?

ocdtrekkie commented 8 years ago

I have no interest in joining Slack. Please just make sure when there's a repository to check out, that the link to it makes it into this thread.

mikaelbr commented 8 years ago

FYI: There's also created a Slack team that can be used for discussing projects from this repo. You can get an automatic invitation here http://opensourceideas.herokuapp.com/

@ocdtrekkie We'll make sure a link to the project is posted here and in this overview 👍

ghost commented 8 years ago

@FutureProg yeah, your email has a spelling mistake previously :)

@ocdtrekkie Sure, we are yet to do a basic setup, we are testing tools and working on the mockup. I need to think about this, I'm keeping slack only for discussing on development. This is the repo https://github.com/mysticmode/libreread I'll notify you about the process in this thread soon.

danfickle commented 8 years ago

Elasticsearch with the Apache Tika mapper plugin already does search of PDFs and many other formats, you just need to provide a web client.

https://hustbill.wordpress.com/2015/09/11/full-text-search-by-elasticsearch-mapper-attachments-in-pdf-format/

ghost commented 7 years ago

Hi, I have created a basic version of LibreRead. You can signup, upload books and it will get indexed in Elastic Search as a background job. Once the indexing is complete, you can search through all books by Metadata and Content. I have implemented PDF.js for UI consistency. Also I'm going to use ePub.js for epubs.

https://github.com/mysticmode/LibreRead

Here are some screenshots of what I built.

There is more work to be done in order to publish this. I think I'm going to roll out the beta version first.

To-Do:

Multiple user Roles
- Super admin (Can manage books and add/revoke users)
- Admin (can upload books)
- User (Only read books)
Book Access
- Public (Any user can read)
- Private (Only the user uploaded can read)
- Permissions (You can select specific users for reading access. This option will be available when you upload)
Create Notes and see other users notes while reading.
ePub implementation

mikaelbr commented 7 years ago

This is well underway with brilliant work from all of you and @mysticmode. I'm closing the issue as it's started and progress has been made. Please feel free to continue the discussion even though the issue is closed.

ghost commented 7 years ago

Hi, I'm asking for code contribution help from you guys for LibreRead.

Major help is needed in the backend which is written in Flask. For setup, Please check the Readme post in the repo for development setup. I can help you with that, see my email below.

I'm working on a new design and it would be greatly helpful if someone could port the old app code to work with the new design and continue taking part with the development.

Current list of features:

Add books (only pdfs)
Full-text search
Collections
Highlights + Annotations

To do list of features:

Multiple user roles
Book Access
ePub Implementation (Major work, Not sure for the first release)

You can email me for further detail hello@nirm.al or please comment here.

ghost commented 7 years ago

UPDATE on LibreRead:

It's a single user product.
Backend being written in Go.
Updated README.md with new features & goals for the initial release.
New design.

Soon I'll show you the version put on the server for testing.

ghost commented 7 years ago

I deployed my code to the server http://172.104.59.151:8080/signin email: rkumarnirmal@gmail.com password: demo

It's not done yet. Right now I'm testing file upload and full-text search. Would love to know your feedback.

Work needs to be done:

Collections
Account settings
Landing page design
Deployment documentation
Much more testing

karuppiah7890 commented 7 years ago

@mysticmode Pretty Neat! I even tried sign up. But in the confirmation mail I got the URL as "localhost:8080...." instead of "172.104.59.151:8080...."

ghost commented 7 years ago

@karuppiah7890 Sorry for that! I should disable signup. It's a single user product :)

ghost commented 7 years ago

Adding one more request. The design is responsive. But still needs some fine-tuning. You could check in hand-held devices and please let me know if I need to make any change.

Thanks!

karuppiah7890 commented 7 years ago

@mysticmode Oh yeah, I remember seeing somewhere in discussions that it's for a single user. Self hosted system and for private use.

karuppiah7890 commented 7 years ago

@mysticmode Not very responsive. Menu is responsive, but books are eaten up (and no scroll bar) when size width decreases. Try decreasing your browser width and see it.

And a good tool to try out responsiveness across lots of devices is : https://sizzy.co/ Code for the sizzy project https://github.com/kitze/sizzy

ghost commented 7 years ago

@karuppiah7890 Could you check in the mobile/tablet browser once? I've added max-device-width

I think that is the reason if you decrease desktop browser width, you are not seeing it responsive.

ghost commented 7 years ago

@karuppiah7890 I could add multi-user support. But that brings more complexity. For example, you wouldn't like your invited user to fill up the storage space with lots of books. We should be providing a way to control the storage space for each user.

And if there is multi-user support, people would expect to share ebooks with other users. In that case, We need to put a notice that we are not responsible for sharing copyrighted ebooks in the platform. And you have to control the book reading access. You wouldn't like to share it to everyone.

Those will take more thoughts and time. I'm trying to keep it simple for the initial release.

karuppiah7890 commented 7 years ago

Makes sense!

karuppiah7890 commented 7 years ago

@mysticmode It works fine in my tab!

karuppiah7890 commented 7 years ago

Cool stuff! :smile:

ghost commented 7 years ago

I've added EPUB support including full-text search feature. But it's still WIP, we need to do more user testing.

For Demo: http://demo.libreread.org/ email: rkumarnirmal@gmail.com password: demo

To Do list:

Test by uploading and using more EPUBs.
Highlights & Annotations for both PDF and EPUB
Account settings

I'm planning to launch beta in the next couple weeks by completing the above to do list.

karuppiah7890 commented 7 years ago

:+1:

ghost commented 7 years ago

IMPORTANT NOTICE: The project repo has been moved to Savannah https://savannah.nongnu.org/projects/libreread/

karuppiah7890 commented 7 years ago

@mysticmode Does the upload work for demo ?

ghost commented 7 years ago

@karuppiah7890 Yeah, is there a problem?

karuppiah7890 commented 7 years ago

Oops. Sorry. Yes, I just noticed. It does work, but it doesn't show the thumbnail of the Ebook though. And I was dumb as I forgot it takes some time to index the PDF in the background and started trying out the search immediately after upload, to see if the book is present as thumbnail wasn't shown. But now I am able to search. Just that the thumbnail is just white

karuppiah7890 commented 7 years ago

The book has a simple page with title (first page) as a thumbnail, from what I see in my file explorer

karuppiah7890 commented 7 years ago

@mysticmode Check the demo site. The book I uploaded is Eloquent JavaScript

ghost commented 7 years ago

Yeah, I can see that and it doesn't generate cover for some PDFs. I'm using poppler-utils to generate PDF cover.

Some days before @ocdtrekkie pointed out that the user should be able to edit the Metadata of the ebook(title, author and cover). So you can add cover to the ebook manually.

I'll be adding that feature in the next coming days :)

karuppiah7890 commented 7 years ago

bug

I see a 404 error for the thumbnail, just noticed it while checking out collections page

karuppiah7890 commented 7 years ago

That sounds cool. It's true. Sometimes the Ebook may not have all the metadata properly. :)

ghost commented 7 years ago

@karuppiah7890 Could you try some other PDFs which has cover image?

As said, I'll add the feature where you could attach PDF cover if it doesn't generate from the code.

karuppiah7890 commented 7 years ago

Yes, it works in a pretty smooth manner

ghost commented 7 years ago

There is a problem with my EPUB implementation. Right now I'm doing it like this

unzip epub
load all htmls into a single file
show that file

That way I could just show the entire epub content as a single html file. I thought scrolling through a single page would be more natural than clicking next and previous buttons.

But I have a problem here, EPUB table of contents doesn't work that way. Each link points to the particular file. I tried to manipulate it, but it didn't go well.

So, I'm going to do EPUB viewer in a traditional way like other EPUB readers. I'm going to use Redis for this.

unzip epub
Fetch the spine data(id and href) and store in redis
Load each html based on the spine data by using next/previous button.

This way table of contents will automatically work and I don't need to do manipulation.

I'll get back when I'm done with this implementation.

ghost commented 7 years ago

Things need to be done for the beta release:

Working EPUB implementation.
Edit/Delete books.
Edit/Delete Collections.
Highlights/Annotations.
Account settings.

ghost commented 7 years ago

EPUB support including the search functionality is implement now. Please let me know your feedback.

http://demo.libreread.org Email: rkumarnirmal@gmail.com Password: demo

I'm going to work on Highlight/Annotations feature.

ghost commented 7 years ago

I have done Highlights & Annotations for PDFs. Now you can Highlight a text and add comment to it on PDF files.

I'm starting to do the same for EPUBs now.

But we need to more user testing on this feature. If you are interested, Please check the uploaded book here http://demo.libreread.org Email: rkumarnirmal@gmail.com Password: demo

Thanks!

karuppiah7890 commented 7 years ago

It's pretty neat @mysticmode ! But I noticed some unusual things while trying out stuff

ghost commented 7 years ago

@karuppiah7890 Thanks for trying it out! What is it?

karuppiah7890 commented 7 years ago

Once I highlight a line and then delete the highlight, then I am not able to highlight it again

ghost commented 7 years ago

@karuppiah7890 Wow! That's a nice find. Thank you! will fix it :)

karuppiah7890 commented 7 years ago

It never gets selected with the blue highlight itself, when trying to highlight a text which is a superset of it. Everything gets selected except the old highlighted text

ghost commented 7 years ago

Ah! That's quite complicated for me now. Highlighting over the already highlighted text. I'll try to do that.

karuppiah7890 commented 7 years ago

No, I mean, highlighting over an already highlighted and deleted highlight text

ghost commented 7 years ago

Yup, I will fix that :)

karuppiah7890 commented 7 years ago

Like, say the text is "This is an example of an highlighted text". I highlight the word "example" and then delete the highlight. Then when I try to highlight whole sentence, it doesn't show blue highlight for "example" while selecting also.

karuppiah7890 commented 7 years ago

And you could change the color of the highlight selection from blue to something else and also make it transparent so people can read when they are highlighting. And I see you use tooltips to show options to change the color and for notes and for deleting highlight. I think you could use the same for asking people if they want to highlight a text - as a tooltip, instead of showing an alert for asking if they want to highlight. I am talking about something like Medium publication highlight feature.

open-source-ideas / ideas