Growing community and namespace collisions

[lordnynex]

@leafo regarding Olivine-Labs/lua-jwt#4

I'm noticing more frequent dependency management issues using luarocks. Right now it seems too easy to break downstream projects from accidental namespace collisions. I'm not sure that the current luarocks implementation creates an opportunity to fix this. The purpose of this issue is to generate a dialog for creating proposal for a more long term fix.

Would you/luarocks be open to the idea of namespace enforcement? For example:

Phase 1 might be to create a static blacklist table to reject rock submissions that declare modules that are known and 'generic' to the community at large? A good example of this is the cjson module. In the last week I've counted 5 different variants. Some implement different functionality but do not have other widely accepted upstream patches. This is causing a lot of harm to the community imho.

Phase 2 might be index declared module namespaces of existing rockspecs and reject submissions that collide. Prompting the developer to be more thoughtful about his/her naming scheme. If outright rejection is not desirable, maybe we can patch luarocks to warn the developer of this conflict before submission? I think this is an easy mistake to make. Other packaging methods (most notably python packages) generally assume that the package name influences the namespace in some way. The differentiation between package and module namespace in luarocks is a bit confusing.

Phase 3 might be to adopt a rockspec naming scheme that mimics github and more recently dockerhub. Something like 'developer/package:version_tag'. This seems beneficial in the long term because it paves the way to adopt semantic versioning as a standard in the lua community and allows developers who are forking existing modules to be clearly differentiated.

These are just off the top of my head. I don't even know if luarocks can support args in the format proposed in phase 3.

[leafo]

I can parse the rockspecs extract the names of the modules they provide. With that I can create a unique index on these names to guarantee there are no conflicts in a single manifest. (Pretty sure the rockspec has to declare all the modules it installs) I think this would be the best first step but I'm sure there are collisions already. This is related to #63

The developer name in front of packages is another interesting idea, but that would also involve rewriting code to be like require("leafo.lapis.application), and I'm sure it won't be easy to get the community to adopt that convention.

[lordnynex]

@leafo Sorry for the delayed response (holiday season).

I've been thinking about this a bit more and I think any initial steps would be beneficial. I agree that it would be extremely burdensome to ask the community to adopt entirely new naming conventions but maybe it would be beneficial to impose incrementally stricter enforcements. I am only just now starting to read about how luarocks works (I've been blindly just installing for years), so the idea may be half baked.

The github style ':developer/package:version' idea was meant to identify an upstream package at install time rather than influence the namespace. The naming convention is inspired by both github and dockerhub. Docker has a very similar problem for user supplied images of community projects; 'ubuntu', for example. I think this may give rocks consumers the ability to 'pin' their dependencies to a known upstream rather than the top level module namespace.

Another possible alternative is to look at the way npm handles this with scoped packages. The python package index has some enforcement in place but I'm not as familiar with it. I think PEP 0382 may be relevant? Without some sort of @username convention It may be beneficial to assign publishing permissions to a namespace on a first come first served basis.

I think a really great addition to luarocks would be the ability to register an account a'la github users/organizations and use the luarocks site more frequently to provide an aggregate view of your work. If you have any of this type of functionality on the roadmap, I'd love to contribute.

Regarding a unique index, I think I have a serializable bloom filter implementation somewhere that may be cut down on your storage costs. I'm not sure how big the tree is, but as it grows index sizes may become burdensome for you to host. Please let me know if I can help at all.

[moteus]

But what about rocks which uses make/cmake/... as build system. There will be problem to detect all names in rockspec. Please look at this discussion (https://github.com/leafo/luarocks-site/issues/19)