search

purescript / pursuit

Website for hosting and searching PureScript API documentation
https://pursuit.purescript.org/
Other
170 stars 47 forks source link

Full text search #318

Open hdgarrood opened 7 years ago

hdgarrood commented 7 years ago

Include comments in the search index, so that e.g. if I search for Kleisli I get Star from Data.Profunctor. Note that the documentation uses the word "Kleisli" but the word "Kleisli" doesn't appear in any of the types, which is why this query currently returns no results.

paf31 commented 7 years ago

I love the idea, but I don't know if we can keep an index in memory, so this might mean using something like ElasticSearch unfortunately.

chexxor commented 7 years ago

Wouldn't need to go all the way to ElasticSearch, do we? I think databases like sqlite and postgres have text-search. sqlite fts Might require adding an option to the search, like "include comments", to activate this extra filter.

Would require moving from filesystem-as-database to sqlite-as-database.

paf31 commented 7 years ago

We could store the index on disk ourselves perhaps. I really like the way we use the filesystem right now, it makes maintenance very simple.

felixSchl commented 7 years ago

How about routinely merging all json files in "data/verified" into a single file containing all the comments? In terms of performance, we either search the file by scanning it or simply keeping it entirely in memory. For the former, we would just do something line-based and for the latter use json.

felixSchl commented 6 years ago

This just showed up in my github feed: https://github.com/well-typed/full-text-search and might be worth looking into. It comes with an example that coincidentally also searches packages.