Tracking issue for custom registry support

MarijnS95 commented 5 months ago

I find myself stuck in the rabbit hole of publishing many crates to a custom registry using cargo-smart-release, and now being a couple crates deep to improve the tooling to make this possible, while also stumbling upon glaring issues in the codebase itself. Let me summarize them here, so that they are not forgotten about.

Config support

[ ] Merge .cargo/config.toml files with precedence rules before parsing the final result (https://doc.rust-lang.org/cargo/reference/config.html);
[ ] Require a replace-with registry to exist, instead of assuming it's unused if [source.your-replacement-registry] doesn't exist;
[ ] Allow finding replace-with keys in the [registries] table as well (https://doc.rust-lang.org/cargo/reference/source-replacement.html);
[ ] Create test cases for various config variations.

Wrapper around sparse or git registry

Following a user URL, user registry name, and eventually a replacement in one of the .cargo/config.toml files, the sparse+ prefix ultimately gets a final say which of the two backends are used. A downstream crate like cargo-smart-release only cares about refreshing and reading information for a specific crate, regardless of which backend is used.

https://github.com/frewsxcv/rust-crates-index/pull/167 is a sort of step-up to disallow accidentally using a sparse+ URL in GitIndex, similar to how SparseIndex requires the URL to start with sparse+http.

Maybe both implementations should also validate, if there already is a folder in .cargo/registry/index/{registry}, that it matches the sparse-ness of the URL?

Constructors with registry name

Since this crate already has support for reading .cargo/config.toml, and downstream crates like cargo-smart-release generally only know the registry name (from --registry <some name> or Cargo.toml's [pacakge] publish = ["custom name"], or even .cargo/config.toml's default registry), it'd be great to extend the from_url() constructors with a from_registry_name() that looks up the URL(s) automatically.

Authentication support for custom registries?

This piggy-backs the above, in my brief searching I haven't found any parsing of token and credential-provider in the [registries.custom-registry] table.

MarijnS95 commented 5 months ago

https://github.com/EmbarkStudios/tame-index it looks like this crate already addresses a bunch of my concerns, though I'd have to look further to see if it solves all of them.

Any clue why there are two competing implementations of the same library, whether one is clearly better, and which one should be used in cargo-smart-release?

Byron commented 5 months ago

The one used by cargo-smart-release was available first. If I remember correctly, tame-index has additional fixes and might generally be better. Both are maintained, but tame-index gets features more actively.

MarijnS95 commented 5 months ago

@Byron that much is clear from the readme in tame-index; it literally starts out with stating everything that is wrong in rust-crates-index. The question is, was there some sort of argument or disagreement that made it infeasible or impossible to improve this crate instead?

This behaviour comes around a lot in the crates ecosystem, and the forker tends to forget how taxing it is on the community to propose an "improved alternative" when the status-quo is also still maintained. Contributing fixes and features directly to the crate that everyone is already using naturally trickles down into consumer crates when releases happen. But instead cargo-smart-release (realistically: my usecase for it, hence my time and mental sanity) now has to investigate whether migrating to tame-index is a better idea than fixing or copy-pasting those into rust-crates-index. And this tracking issue should be indicative of just how much is missing/broken.

Since you @Byron seem to be the only maintainer of rust-crates-index, would migrating cargo-smart-release to tame-index (assuming it is clearly superior, and doesn't miss any features/fixes from this crate) be a reason to cease maintainership and archive this crate?

crates.io indicates that both crates get almost equal downloads lately, so the community is clearly undecided: https://crates.io/crates/crates-index https://crates.io/crates/tame-index

And it seems that you have been the one adding tame-index to the README here, clearly stating that it is a hard fork with many improvements and advantages.

Byron commented 5 months ago

@Byron that much is clear from the readme in tame-index; it literally starts out with stating everything that is wrong in rust-crates-index. The question is, was there some sort of argument or disagreement that made it infeasible or impossible to improve this crate instead?

I have no information about that. My guess is that they had to move fast, which is easiest with a fork.

Since you @Byron seem to be the only maintainer of rust-crates-index, would migrating cargo-smart-release to tame-index (assuming it is clearly superior, and doesn't miss any features/fixes from this crate) be a reason to cease maintainership and archive this crate?

I am just an external maintainer and don't have control over the repository. If I'd quit, somebody else might take over as it's still quite a widely used crate. Unifying both across the ecosystem is probably out-of-scope for this endeavour, and independent of what cargo-smart-release ends up using.

And it seems that you have been the one adding tame-index to the README here, clearly stating that it is a hard fork with many improvements and advantages.

It's a great crate and there is nothing wrong using it. If it provides a better starting position to get the features needed for cargo-smart-release, trying to get them there might be a good play. It depends on how much time the author has to keep maintaining it.

MarijnS95 commented 5 months ago

My guess is that they had to move fast, which is easiest with a fork.

Which is a really poor excuse in my opinion, when a soft-fork would (temporarily) suffice. Especially considering what you say after:

It depends on how much time the author has to keep maintaining it.

Which means it has been a waste of time for everyone involved in the end, if it both crates end up in some stale half-(un)maintained state.

But alas, I digress, that's all for expressing my general dissatisfaction in the crates ecosystem "lately". This crate is just one of many, many examples.

Unifying both across the ecosystem is probably out-of-scope for this endeavour, and independent of what cargo-smart-release ends up using.

Not something I'd be interested in burning my hands on, though it does make it harder to pick one over the other. https://crates.io/crates/cargo-smart-release doesn't seem like it will make a dent in the downloads count for rust-crates-index.

I'll queue this up on "my list" to see if tame-index can solve part of the issues in cargo-smart-release and let you know.

MarijnS95 commented 5 months ago

For reference, I've started checking how much of the above points have already been addressed in tame-index. Browsing the source-code, automatically turning a registry name into its URL (by parsing .cargo/config.toml and special env vars) does not seem to be implemented yet, so I'm tracking that in https://github.com/EmbarkStudios/tame-index/issues/60.

frewsxcv / rust-crates-index