search

fabd / kanji-koohii

A web application to help Japanese language learners remember the kanji.
https://kanji.koohii.com
GNU Affero General Public License v3.0
228 stars 21 forks source link

Allow story search #110

Open nicolasmaia opened 7 years ago

nicolasmaia commented 7 years ago

It would be great if people could search terms inside stories. The other day I found a Samurai Jack-themed story and wanted to find more, but couldn't :T

fabd commented 7 years ago

I agree it's a good idea. Functonally similar to allowing searching for primitives. Not sure how the UI would be presented so as to not cause confusion between kanji search, and story search.

Implementation concerns

I will tag it for now as "help wanted" however because I'm not sure how to do this the right way in MySQL.

The stories table is BIG, nearly 1 GB. And the queries are already quite slow. Some of my solutions didn't work on the web host, such as using MySQL partitioning.

Search could work just fine, or make the database even slower I don't know... it could also take a lot of additional space on the db.

I would need advice from someone experienced in SQL here. So currently on hold.

nicolasmaia commented 7 years ago

In terms of UI, the search field could probably be nested between the menu icon and the heart icon.

Thanks for considering it. Let's hope a SQL wizard come this way.

iveskins commented 7 years ago

Currently you can see some stories this by google searching: site:https://hochanh.github.io/rtk/ "jack" But that site only displays a few stories.

kanji.koohii.com stories are not indexed by Google it seems.

fabd commented 7 years ago

Currently you can see some stories this by google searching:

This site has a DMCA takedown, I am guessing (I don't know for sure) that the RTK publishers complained to github as I think it featured the copyrighted stories from Heisig as well.

It certainly would have been nice of the author to contact me before pulling koohii stories. 😠

kanji.koohii.com stories are not indexed by Google it seems.

There is a good reason why stories are not visible to guests (and hence, not indexed by Google).

Kanji stories are highly imaginative, and vary wildly in their content. The way Google works, a lot of people would find their way to the Kanji stories searching for terms related to policitcs, religion, ..; and many many topics both scientific, completely unscientific, controversial, and so on.

Koohii users understand they are mnemonics but random visitors won't. I will likely get complaints from people who don't understand it's all imagination. I will also make the search experience worse for people who are not interested in kanji stories but somehow find their way to the mnemonics due to search terms.

Google is not smart enough to understand these are not mnemonics. At best it would recognize the Study pages as covering a wild array of topics.

Worst case, it would get the site flagged for spam and other things. So I'd rather be on the safe side.

And of course if we implement a search eventually, I don't think there is much to be gained by Google indexation of these pages (besides that Google search is of course superior to most builtin website searches).

nicolasmaia commented 6 years ago

I thought a bit about this and I think the UI could take a hint from how Wikipedia's search box looks like:

captura de tela de 2018-04-07 18-36-55

So you have title matches and below, you can click to search the body of the pages.

brianbv commented 5 years ago

I would need advice from someone experienced in SQL here. So currently on hold.

I think you would be able to use a full-text index if you aren't already. It will make the database bigger, not sure by how much though.

https://dev.mysql.com/doc/refman/5.6/en/fulltext-search.html