Proposal: Modular standard library and crates.io improvements

[SmushyTaco]

Rust has a small standard library because they don't want to add something bad to the standard library because if they do they won't fix or remove it because it would break compatibility. I've been thinking of a solution and this is what I've come up with:

Before I explain my modular standard library idea I need to explain my crates.io improvements:

1. There will be crate titles for crates with the default title being unverified. If a crate is verified it means there's nothing malicious about it. If a crate is official it means it's maintained by the official Rust team. If a crate is unverified it means the crate isn't gaurenteed to be safe.
2. Precompiled crates for efficiency.
3. Crate groups, let's say somebody wants to add a ton of packages as dependencies for their project but they don't want all of it cluttering their cargo.toml. You'd go onto crates.io and create a crate group with all the specified crates. Multiple versions of these groups can be made like with normal creates (so 1.0.0, 1.2.4, 2.0.5, etc).

Now with all that out of the way this is my proposal which would allow the Rust standard library to be big with the freedom to make breaking changes without breaking compatibility:

Make every little part of the standard library a crate and have the standard library be a crate group that's added by default. This way there can be different versions of every module that's part of the standard library so breaking changes can be made and corrected without breaking anything because developers could simply use an older version of that specific crate from the standard library.

[tesuji]

they do they won't fix or remove it because it would break compatibility

It is not just that, adding more things make std maintainers life harder. You have more works to do and take care of. It's mostly boiled down to resources. If Rust org has more resources (including people), they could do a few things mentioned.

In short, these ideas has been proposed in some posts/media, but never have enough resources to do.

[Lokathor]

If we can't even ship pre-built quote and syn to keep proc-macro build times down, I don't know how we'll ever manage to ship pre-built anything else to keep build times down.

[SmushyTaco]

they do they won't fix or remove it because it would break compatibility

It is not just that, adding more things make std maintainers life harder.

You have more works to do and take care of.

It's mostly boiled down to resources. If Rust org has more resources (including people),

they could do a few things mentioned.

In short, these ideas has been proposed in some posts/media, but never have

enough resources to do.

Even if the team doesn't have the resources for a big standard library these changes should still be made because mistakes can still appear in a small standard library and this system is the best system to correct them.

[SmushyTaco]

If we can't even ship pre-built quote and syn to keep proc-macro build times down, I don't know how we'll ever manage to ship pre-built anything else to keep build times down.

Aw that sucks to hear, well scratch what I said about precompiled crates then.

[tesuji]

these changes should still be made because mistakes can still appear in a small standard library and this system is the best system to correct them

I don't want to be discourage. But there are things easier said than done. Who would maintain those crates? Who review them? What about qualities? Do they need to be built/passed CI? Who implement initial features to classify crates/verify it and maintain it when it has issues?

[SmushyTaco]

these changes should still be made because mistakes can still appear in a small standard library and this system is the best system to correct them

I don't want to be discourage. But there are things easier said than done.

Who would maintain those crates? Who review them? What about qualities? Do they need to be built/passed CI?

Who implement initial features to classify crates/verify it and maintain it when it has issues?

The whole idea is to make the standard library modular, it's a different structure but the same people who maintain the standard library would maintain the standard library.

I'm not sure who would implement this new crate system but this is all open source so I assume the Rust community would contribute to this if it gains enough support.

Regarding verifying crates, a review team for that would need to be made which would also probably be ran by the community if it gains enough support.

[tesuji]

The whole idea is to make the standard library modular, it's a different structure but the same people who maintain the standard library would maintain the standard library.

And that's the problem. They don't have bandwidth, but they want to keep std high quality. They attend FCPs to decide what going to stable. I know some members in std lib team don't want to be assigned for reviewing PRs. Some are busy in working group, for example inline-asm, errror-handling, simd, etc... Some at works.

Also, not all are expert in every domains, like GUI or crypto. We don't have people for that.

[SmushyTaco]

The whole idea is to make the standard library modular, it's a different structure but the same people who maintain the standard library would maintain the standard library.

And that's the problem. They don't have bandwidth, but they want to keep std high quality.

They can only attend FCPs to decide what going to stable.

I know some members in std lib team don't want to be assigned for reviewing PRs.

Some are busy in working group, for example inline-asm, errror-handling, simd, etc...

Some at works.

Also, not all are expert in every domains, like GUI or crypto. We don't have people for that.

If you have an actual critique about my proposal I'd love to hear it but all I've heard you say about it is that it won't work because nobody is probably going to work on it which seems like an irrelevant point to bring up because things like the support something has, the resources there are to implement it, and the people who are willing to contribute can change. I'm not saying this needs to be implemented right now, I'm just saying it should be implemented eventually. Is there anything you don't like about the idea itself or is weather it's feasible your only concern?

[lebensterben]

Breaking STD into pieces and even allowing people to use use arbitrary combinations of those pieces sounds like a nightmare. How are you supposed to define your minimum rust version policy? Like some parts require version XXX some other YYY? No thanks.

[SmushyTaco]

Breaking STD into pieces and even allowing people to use use arbitrary combinations of those pieces sounds like a nightmare. How are you supposed to define your minimum rust version policy? Like some parts require version XXX some other YYY? No thanks.

Well every crate from the standard library would depend on whatever it needs to work, so if a specific version of x needs version 2 of y and version 3 of z to run it would use that because these dependencies would be specified for that crate (Just like it is with any other crate).

[chb0github]

This is a classic version / dependency problem. The original Java libraries years ago shipped with the JDK because there really was no other means at the time to have a bundle of verified, but dependent code - In 1994 Java was already becoming a popular language and the internet as we know it was just becoming public. Did you know the core Java UI libraries ship by default along with all the backend libraries (up until java 9)? What interest does a webserver have in desktop UI widgets?

This all resulted in version release cycles in the order of 5-6 years with dozens of features and hundreds of bug fixes. Basically, a monolith.

A few years ago Java finally released a modular system for all dependencies and they have been slicing off core libraries into separate modules ever since. In addition, the core maintainers finally realized that a 6+ year release cycle was making Java look like an irrelevant dinosaur. So, they open-sourced it and it's on a 6 month release cycle.

While platforms like .NET core and nodejs come stripped down, this has the upside of relieving maintainers from improving anything but core language features. The flip side is, of course, to some degree, it becomes the wild-wild-west when trying to get basic things done because you have to find a dependent library to do the job.

As for complaints of "but who would maintain it?!" and "official" packages: Making dependencies 100% modular but then shipping with a default bundle in no way impedes support ability - it in fact, enhances it. People are always going to take the defaults unless they have a compelling reason not to (laziness will win everytime).

As for the complaints about which version goes with what: That's called software development; you should always test when any dependency (OS, processor, etc.) changes. This is no different. I like the idea of bundling (internally at Amazon, they do exactly this and it's called a "Version Set").

I would suggest that research be done on how the dozens of other languages around the world have managed dependencies and what issues they have faced - this isn't even a remotely new problem.