search

redex / redex.github.io

Reason package index
https://redex.github.io/
MIT License
65 stars 18 forks source link

Also using the Yarn index for data source #13

Closed Haroenv closed 6 years ago

Haroenv commented 6 years ago

Hey! We're indexing everything on npm on the index called npm-search for https://yarnpkg.com. I was wondering if you could have any use in this index, it might have things you find useful. You can refine for the specific parts that you need to augment your own index, browse and then do what you want (since some aren't on npm).

Let me know if you're interested and I can see how I can help

glennsl commented 6 years ago

Hey, that's cool! I was not aware of yarnpkg search, and I'm surprised at how similar it looks. Particularly the license "badge" is eerily similar :)

Anyway, I pull data primarily from https://npms.io today, which does some analysis that yields a set of scores that I find very useful. Unfortunately, some of those scores are normalized against the entirety of npm, which doesn't work out so well for the comparatively small Reason ecosystem. So we're considering doing (part of) that analysis ourselves. Even if we do all of it ourselves it still seems easier to consume the npms API since we can fetch them one by one (see https://api.npms.io/v2/package/algoliasearch for example). I also don't see npm-search having anything npms does not (it does have a nicer format for some things though).

Let me know if I've missed something though.

I did find some great tips for improving the ranking while looking into this, but there's one issue I still haven't been able to solve. Penalizing flagged (i.e. deprecated or neglected) results. I'd still like them to show up in the search results, but at the bottom as a last resort. Do you know if it's possible to achieve that?

Haroenv commented 6 years ago

Something it has which npms doesn’t is “owner” which is owner on the repository if it’s available or as fallback on npm. This is nice for bigger organisations.

You can use getObject because it’s an Algolia index, or filter by objectID.

It also has some things slightly different regarding readme (completely) and the url for the changelog.

Haroenv commented 6 years ago

It will also be faster than npms because it’s an Algolia index.

glennsl commented 6 years ago

We're moving towards using scoped packages, but it'd definitely be a good idea to fall back to owner when that's missing. And repo owner is much better than whoever happens to have publishing rights on npm.

Changelog might also be interesting in the future. Right now most people just put it at the bottom of the readme though.

I'll think and look into this a bit more. For testing I can just use the api key from yarnpkg, right? Then when we get serious it'd probably be better if we have our own key.

Thanks for getting in touch!

Haroenv commented 6 years ago

Yep, send am email to support@algolia.com and mention me and yarn and we’ll get a proper api key when you are done testing