Scoped package support

[seldo]

Scoped packages can be public or private, so to support them in the downloads API we need an authentication mechanism that can, at minimum, throw a 404 if the package is private, but ideally provide stats for private packages to authorized users.

[kgryte]

+1.

[joshrtay]

where is this at?

[zackify]

Would love to see stats for public scoped packages! :)

[justinfagnani]

FYI, the Polymer team is looking at publishing > 100 packages to a scope, but we really need them to show up in search.

[soyuka]

+1 about search, https://github.com/npm/newww/issues/929 mentioned that scope packages don't end up in the search but this issue feels like it's all about statistics. IMO search feature priority is greater than statistics :).

[justinfagnani]

It looks like scoped packages don't show up in API queries either.

This query: https://skimdb.npmjs.com/registry/_design/app/_view/byKeyword?startkey=%5B%22polymer%22%5D&endkey=%5B%22polymer%22,%7B%7D%5D&group_level=3

Doesn't return @polymer's scoped packages that have the 'polymer' keyword.

[zenorocha]

+1

[ebidel]

+1

[zackify]

is this ever going to happen... lol

[cheapsteak]

It's been 10 months and only 8 people have complained about it, I suppose that would make something low-priority

[justinfagnani]

Maybe because without search support, people won't use scoped packages much. We have > 100 scoped packages now, but haven't advertised then because of this.

[kgryte]

@cheapsteak This is incorrect. Scoped package support has been a recurring issue both here and for www. As alluded to by @justinfagnani, lack of search support removes an incentive for people to use scoped packages, which was also mentioned elsewhere by @ashleygwilliams. For example, I only use scoped packages for modules for which I don't care if they show in search, such as personal build tools, lint configuration files, etc. For modules which are "public", I intentionally avoid scoped packages, which, because they do not even show with a keyword search, cannot be semantically "bundled" as disparate components.

[cheapsteak]

@kgryte I should have mentioned that want it as much as you guys do and have been subscribed to this thread for a few months now

However, looking at it from npm's perspective (I'm not affiliated with them btw, only guessing), we are trying to petition a corporation to allocate resources to solve a problem. I'm sure this is on their list of todos, but how much priority can they give to an issue that only seems to affect, however deeply, 9 people/orgs? The majority of the users of npm wouldn't care whether they're typing npm install package or npm install @org/package, and I'm guessing most organizations that pay for accounts are paying to keep their modules private and won't care about search or stats, or if they do, they're keeping rather quiet.

This might also be related to Github's problem of only being able to have people show support for an issue by posting a "+1", and there might be some who are too polite to post a "+1" that are not included in this headcount, but then again look at how long it's been and what it took for Github to start looking into fixing that

[subfuzion]

It's really a shame since there is so much value in being able to search for packages and see stats to give you a sense of how solid a package is. The community relies on that too much to give up. I get that it's not a priority to fix this ("lots of moving parts, it needs to be done right," etc), but I was disappointed last May and have since abandoned the idea of using scoped packages. I suppose this comment really belongs at https://github.com/npm/newww/issues/929, but, oh well, the thread here drew me in.

[justinfagnani]

Maybe @rockbot can comment. Are scoped packages really intended to be supported? Did we mess up by publishing to scopes?

[drew-walker]

I would really love to see this implemented as well. We're starting to use orgs and teams more heavily at our company, but not being able to see download stats for packages makes it hard to form a business case around more extensive use of npmjs.

[justinfagnani]

@seldo any idea on the status of scoped packages?

[seldo]

We are currently designing an authentication mechanism that would allow us to have APIs that cover private modules. That would remove the barrier to the downloads API covering scoped modules.

[seldo]

(And by "designing an authentication mechanism" I mean "figuring out OAuth, I guess, dammit")

[justinfagnani]

@seldo What do private modules have to do with this? We want public search to work on public scoped packages. See https://www.npmjs.com/~polymer None of those packages show up in search.

[seldo]

Scoped packages can be public or private, and can switch from public to private. To provide stats on them without leaking information, the stats API needs to be aware of the ACL restrictions.

Obviously, once the package is public you won't need auth to get stats on them, but the stats API needs to know.

[justinfagnani]

Thanks for the explanation. Couldn't the existing APIs only search public packages without first designing the auth system? Then only private packages are blocked, not all scoped packages.

[seldo]

You're right that there's a shortcut that doesn't involve designing auth to allow access to public packages only. We'd still need to build something that doesn't currently exist -- a connection from the stats API to the ACL, and of course we'd need to start counting those stats, which the current system doesn't do.

I don't have any time or engineers to put on this in the short term, unfortunately.

[kgryte]

@justinfagnani I am not certain that search really helps here. For instance, if I perform a keyword search https://www.npmjs.com/browse/keyword/polymer, the scoped @polymer packages do not appear until page 4 => https://www.npmjs.com/browse/keyword/polymer?offset=109. Even if scoped packages appeared more readily in search (currently, you actually need to click on a keyword), scoped packages will still be returned as part of a much larger subset, which for various reasons may rank higher. The reality is search should be a minimal factor in determining whether to use scoped packages. Even with better search support, users will not be overwhelmingly better off.

Further, other people in this thread have mentioned stats as a blocker. If people base their decisions on stats (downloads, stars, etc), they are making poor decisions. Stats are a very weak proxy for module quality. Fortunately and unfortunately, the only reliable way to evaluate quality is by reading the source.

The primary reason to use scoped packages is the ability to control a namespace, period. And scoped packages already support this use case. Whether or not scoped packages have easily accessible stats or show up first in search results is minor in comparison. The best option for directing users to your scoped packages is by creating your own hosted documentation which aggregates your scoped packages and shows how those packages are part of a coherent whole. No matter what level of support NPM provides, it won't satisfy this use case, nor should it have to.

[subfuzion]

@kgryte I couldn't disagree more -- not that browsing the source isn't a good thing to do, but stats are a valuable part of the decision making process when searching and evaluating published packages. Stats help determine the usefulness of a package as evidenced by the number of packages that depend on it, which contribute to the download count. Determining the momentum and activity behind a healthy, thriving project also helps, which you can do by checking GitHub issues, pull requests, latest commits, etc. Open source contributors also tend to find stats to be not only useful but gratifying. Perhaps it's not important to you, but you certainly don't speak for everyone. We all get the namespace argument, but I think we'll find scoped packages to be much more popular and published more frequently when stats are supported.

[fvdm]

I agree with @subfuzion

When it comes to quality, sure a hit counter (read: popularity) won't say much, at least not really to anyone other than the publisher. For quick quality checks we have public CI tests and status badges in the readme, and the source code. However, that same statistic does have value to the developers who published the package or contributed. It gives an impression of how often the code is used, is it being used more now then earlier (using sites like npm-stat.com), is it dead, etc.

I recently published a scoped package only to avoid using someone elses name, but without stats it feels like I'm left in the dark. I have totally no idea if anyone actually installed the package. Of course I could republish it under a different name, but then I have to maintain both of them since I still don't know if anyone is depending on it in their private code.

[RyanCavanaugh]

We're going to be using scoped packages pretty broadly soon and download counts would be an extremely useful thing for us to have. FWIW we have no intention of ever having a private package under the scope in question.

[justinfagnani]

@RyanCavanaugh I totally encourage you to use scopes, since hopefully that'll light a fire on this issue (and it's the right solution for large orgs or packages, IMO), but fair warning that it's more than stats that aren't supported. Notably scoped packages don't show up at all in search (though they apparently do via keyword browsing).

[drew-walker]

Unfortunately we've now moved away from using scoped packages until the search and stats are available. Without knowing how frequently packages are downloaded it's difficult for us to decide which packages we should invest our time in developing further.

[RyanCavanaugh]

@justinfagnani good caveats to keep in mind. We're also somewhat annoyed by the lack of search support, but understand the trade-off there and will be mitigating that with a separate solution.

[rickschmoo]

Any progress on this? We would love some stats for our scoped (public) packages.

[seldo]

Work has been going towards this for the last couple of weeks as part of a larger project. Still no time of arrival I can promise but it's not back-burnered any more.

[RyanCavanaugh]

Just seeing if there's any news on this. Even if the data is only available through some API, we'd love to be able to see it.

[RyanCavanaugh]

@seldo we took some substantial bets on scoped packages on the belief that this data would be eventually available in some form or another. If you could let us know one way or another if this is actually happening (i.e. in the next few months or so), it would be really useful - we need to prioritize some work differently our end depending on which way that goes.

[zackify]

Okay, so now there are public organizations, any progress on this? Would really love to see stats on things I have to publish under my work orgs.

[seldo]

A new API supporting downloads for scoped packages is in the final stages of testing. Expect movement this week.

[seldo]

It shipped! 🎉🎉🎉🎉🎉

Huge thanks to @bcoe for getting this done.

[kgryte]

Thanks @bcoe and @seldo!