Uniqueness on which context?

[tomas]

So according to the shards.yml spec, all package names must be unique. However there's no mention of an official registry to see which names have been registered and which are available.

Or does it simply mean unique in the context of the dependencies installed locally? If that's the case I would suggest a better wording to make it clearer. Like Crystal clear. ;)

[bcardiff]

They must be unique in the context of the shards.yml . There is no global registry for crystal packages. The intention is for them to be descentralized.

This description is in the context of the shards.yml spec. I would think is clear enough but feel free to PR a better wording :-)

[tomas]

OK, will do.

Now, when you say "the intention" is it because this topic hasn't been settled yet?

I think this should be as clear as possible, considering there already are two websites showcasing Crystal shards.

[ysbaddaden]

Shards is decentralized. There is no central registry that we talk to, and there is no plan (in the foreseeable future) for Shards to ever resolve dependencies using such a registry. Shards' names are thus unique in the context of a shard.yml.

Still, the shard name must be unique, otherwise adding a dependency may eventually generate a conflict that Shards will be incapable to handle, more so to install.

An informational registry will eventually happen to ease the pain of searching for unique names.

[tomas]

@ysbaddaden I understand the reasoning and I fully embrace the idea of a non-centralized package system. However the idea of requiring package names to be unique without a proper way of enforcing it seems problematic.

I'm fine with any decision, by the way. Just pointing out that this is a crucial part of Crystal as an ecosystem, and thus shouldn't be treated lightly. I personally think that for names to be unique there needs to be a registry where packages are published (a-la Rubygems, npm, etc). If that's not the case then there's no way of really knowing whether a package is unique or not, and thus the requirement seems kind of pointless.

Go seems to have resolved this, by the way, although I don't know much about the Go ecosystem myself. :)

[asterite]

Well, here's a good reason to have a centralized repository: https://www.reddit.com/r/programming/comments/7olf5e/npm_registry_many_packages_are_disappearing_all/

If I have a popular crystal repo in github and I decide to remove it, good luck with, I just broke your code.

Of course you can have it cached somewhere. We download dependencies to lib, but these are not committed, so prone to be lost. If we had a centralized registry we could make sure that these don't get deleted (unlike what npm does).

This would also solve the uniqueness problem: if it's in the registry, it's taken.

As far as I remember, the only two reasons not to have this centralized is to avoid a shard called mysql2 because mysql was already taken but left un maintained (or is worse). The other reason is not having to pay a server (and code that logic). But these reasons are not good enough. And a centralized server also means we can compute a hash of all the code and check that when downloading stuff. Plus we can have a manifest of all shards and versions to compute dependencies faster. It also means you could run shards add kemal and have it work, or search shards with the shard command, and so on.

[RX14]

@asterite but then we are vulnerable to what just happened: things getting deleted en-masse from the centralized respository. I am strongly against a centralised source of trust regarding shards.

If someone decides to delete their git repository, sure, you just broke people's build, but you're not causing a security vulnerability. The chances of a very commonly used shard getting it's repo deleted is probably quite small.

We could also pin gpg signing keys directly in shard.yml, providing an extra layer of security.

[asterite]

The centralized repository won't allow deleting stuff. It's simple it will keep copies of shards forever.

You say chances are small, but it already happened twice in npm.

[RX14]

@asterite the NPM issue was surely packages being deleted from the DB, which a registry would be vulnerable to regardless of whether it appered to users to be immutable or not.

[asterite]

I think it happened because you are allowed to remove packages from npm not an accident.

In any case, without a registry the problem of uniqueness still exists. I thought one could simply search "crystal shard somename" to know if a name is taken, but it seems that doesn't always work, so there's a risk of collision. Of course, since we are all connected through internet, it's just a matter of discussing which name stays and which one gets renamed (like what happened with brew install crystal)

[RX14]

@asterite the wording was "106 packages were accidentally removed from npm, 96 of which were restored". That seems very unlikely to be users deleting these packages.

I am very strongly against any centralization of shards.

[straight-shoota]

This problem of a repos vanishing can be mitigated by having a mirror which can still provided if the main source fails. This doesn't need a cerntralized repository but can be equally decentralized (speak: many mirrors, either for individual repos or a compiled list of the most popular ones). Since this is particularly relevant for popular shards it shouldn't be to difficult to have such a one available in a mirror. The only question is how shards/the user would be made aware of such a mirror. But it could be a just a list of mirrors in the shard.yml (or a system global config). If shards fails to load a shard from it's primary source, it can look if any of the mirrors provides it and either install it directly or suggest it and ask if it should be installed from a different source.