search is really annoying

[ghost]

I've already mentioned this but I'm creating an issue here to invite some discussion about how npmd search should work. I used to use search a ton back when it was just a faster version of npm search but now it's pretty much unusable because it:

• truncates results after the first page (never ever what I want, particularly when too many things match)
• matches text from the readme and folds the matches into the results (way too spammy)
• matches too much

I really just want the old functionality and feel like it shouldn't be in npmd's scope to invent a new kind of search. I find that I've even started going back to npm search which is crazy slow because npmd search is so bad. Experimentation into new ways of doing searches seems like it would best be done in a separate module?

[dominictarr]

hmm, I find the results useful, but I never used the old search, but anyway, it would be easy to add an option to work like the old search.

many people complain about the search/module discovery, and npmd has all the data. experimenting with search was part of the npmd plan since the start,

but I have no problem being able to turn all that stuff off.

can you put in a pull request?

what do you think about dumping the results into less if stdout is TTY? then you'll have the best results at the top, which is what I was going for by limiting results to one page.

[dominictarr]

okay, this was low hanging fruit, and I was pretty sure I had this implemented already, so I just fixed it.

install npmd@0.19.1 and use the following command line args

npmd search doge --results --no-show-readme

or put this in your .npmdrc

echo '{"results":99999, "showReadme":false}' > ~/.npmdrc

[ghost]

Excellent, will do!

[ghost]

also posting my stream of consciousness from IRC about my thoughts about search in case some other people want to jump in with ideas:

18:35 < substack> dominictarr_: so one of the big problems with truncating results is that the 
                  search is ALSO really over-active
18:35 < substack> so you get SO MUCH spam
18:35 < substack> and all the results on the first (and only) page are useless
18:36 < substack> having one of the terms in the name should count for 1000x
18:36 < substack> and the desc should count for 100x
18:36 < substack> and having a term in the readme should count for 1x
18:37 < substack> part of the reason for this is how modules should do exactly 1 thing well
18:37 < substack> and if they are doing lots of unrelated shit and stuffing that into the readme 
                  somewhere I don't even want to know about it
18:37 < substack> because I just won't use that module
18:38 < substack> I don't actually think that the finding modules problem is related to textual 
                  search
18:38 < substack> it's much more about the network effects of using modules in a style that you 
                  like or written by people who you already trust
18:39 < substack> the full text index seems mostly just like a distraction