crystal-lang / shards

Dependency manager for the Crystal language
Other
758 stars 99 forks source link

Resilience for shard sources #601

Open straight-shoota opened 7 months ago

straight-shoota commented 7 months ago

Shards is decentral by design and thus systematically resiliant against a single point of failure. In practice, most publicly available shards are hosted on GitHub, which thus acts as a single part of failure for a big portion of the ecosystem. So there's quite a big point of failure for that. One might argue that GitHub is too big too fail, and maybe there's some truth to that. But even GitHub has had some temporal outages in the past.

Regardless of GitHub's prominent role: any hoster can temporally (or permanently) drop out. If that happens, shards install won't be able to fetch shards hosted there. If the shard source is still available somewhere else, you could still make it work somehow, for example with manual overrides to the dependency location in shard.yml or shard.override.yml. This requires explict interaction and knowledge about mirror repositories. So it's not ideal.

It's generally a good idea to have libraries available on multiple hosters. This improves resilience against provider losses.

Maybe we could find a way to include information about repository mirros in the shard metadata? I was thinking about the repository property of shard.yml: It could allow multiple values, pointing to alternative sources for the shard. shards could use that information when fetching the dependency.

Of course, this is a bit of a problem because when you don't have the dependency installed yet, shards could't know that its shard.yml announces alternative sources. The information would be available when the dependency is already installed (e.g. for shards update). We could also consider to store such metadata about dependencies in a separte location that's checked into source control of the dependee. Maybe shard.lock could be enhanced to hold this kind of information?

And of course with a structured definition for repository metadata, shard indices such as https://shardbox.org or https://shards.info could make it publicly available which would survive the loss of a single hoster.