Improving binary distribution through cargo.

[XAMPPRocky]

Summary / Motivation

The current cargo-install user experience is very poor. For both the user's (people using cargo-install) and the developer's (people publishing binaries through cargo-publish). For example having no upgrade story, having no way to separate dependencies between libraries and binaries, and having no awareness of binaries on the crates.io website.

CLI applications are currently the third most popular category on crates.io. From general purpose tools like ripgrep, tokei, and hyperfine, to rust development tools like the [diesel CLI], to the myriad of cargo plugins which one of the biggest categories on crates.io. Despite crates.io initially only being designed for libraries it's clear that people want to use it to distribute and share Rust applications.

Cargo and the crates.io is a lot of developer's first distribution of their CLI application. Improving this experience helps smooth out building a CLI application to rust by having a complete distribution experience out of the box with Cargo.

Upgrading

The is no current way in cargo to upgrade a binary. The user's options are to run cargo install --force <package> or cargo uninstall <package> && cargo install <package>. The problem with both of these options is that neither respects the currently installed package's version so to upgrade a cargo package requires a complete reinstall of the application regardless of whether there is a newer version.

This is a very poor experience as Rust developer's we know how long Rust release compile times are. Even with with all the great work the compiler does to reduce compile times there is no better compile time than having to not compile at all. Cargo should provide upgrade functionality either through a new subcommand a la brew upgrade <package> or through a flag like pip pip install -u <package>.

Separation of dependencies

Currently cargo allows you to specify the developer's intent of a dependency with [dev-dependencies] allowing a developer to use different dependencies for development that the end user of their library or application doesn't need to download or worry about compiling for their system. cargo-install does not respect this and will download and build these packages.

Cargo currently does not have the ability to separate dependencies for binaries and libraries. While at first this doesn't seem like a big issue since a binary that depends on the library has to get all of their dependencies. This becomes a problem for the reverse libraries downloading their binary's dependencies. Which becomes an issue when installing a CLI through cargo-install.

Cargo should provide a way to specify library or binary specific dependencies. This could be done in a similar way to development dependencies and having [lib-dependencies], [bin.dependencies], and [bin.bin_name.dependencies] which would allow developer's specify which packages are used only for libraries or binaries while retaining the current behaviour and having [dependencies] be for crates that used in all.

It might not be a big deal to get these packages but they do add a non trivial amount of compile time. We also should not be wasting a user's bandwidth there is still a lot of situations where bandwidth is tight whether the user lives in an area with poor service or is using cellular which are no where near the level's of bandwidth or stability, so any reduction of data is a big improvement.

Showcasing applications

Currently crates.io only shows instructions on how to add a crate as library dependency. Cargo should show cargo install <package> for binary only, and show other binaries that are part of that crate.

Unresolved questions

• How cargo would remember features of an installed app?

[epage]

Providing an easy upgrade story for users.

I'm unsure if cargo-install-update offers this but one thing I frequently run into for CIs is the need to only install if the current version is different than my desired version.

Granted, having prebuilt tarballs and using trust is much preferred.

[epage]

Kind of like our defining what is in scope for this working group by defining a CLI, I think it'd be important for us to define what we think is the scope of cargo install.

One way of doing this is to evaluate what are our packaging and distribution requirements, consider what the ideal solutions are (like my CI use case is solved better if its easy to make prebuilt binaries), and then see what is left for cargo install.

[XAMPPRocky]

@epage cargo-install-update does provide that as far as I can tell.

One way of doing this is to evaluate what are our packaging and distribution requirements, consider what the ideal solutions are (like my CI use case is solved better if its easy to make prebuilt binaries), and then see what is left for cargo install.

I thought with separating out this issue into its own we'd remove the idea that building out packaging and distribution comes at odds with building out cargo-install. As far I'm concerned we should be building out cargo-install like there are no other package managers as it's the only one that we control and is the package manager a lot of rust developers become familiar with when they are building their apps. Just like how nearly all JavaScript clis can be installed with npm install -g I want nearly all Rust clis available through cargo-install.

[epage]

I thought with separating out this issue into its own we'd remove the idea that building out packaging and distribution comes at odds with building out cargo-install

Ok, I thought the value of splitting this out from #8 was because of how different the overall discussions would be. #8 is about documentation, tooling, package formats to support, and policies related to those formats. A discussion on cargo install is more about requirements gathering, feature design, and implementation.

like there are no other package managers as it's ... the package manager a lot of rust developers become familiar with when they are building their apps

1. Since cargo install requires having rust installed, it feels like it'd be off putting to most non-rust users. I know I skip any github project that uses npm install (and not just because I avoid node/electron). So I feel Rust developers need to be willing to consider OS-specific distribution methods unless the app is targeted only at Rust developers (like extensions to cargo). Hopefully we make this dead simple in #8.

2. imo the requirements for distributing source and distributing binaries are so drastically different that we're effectively creating a new packaging scheme that Rust developers will need to know with little reuse of the existing one. We'll also be having to reimplement a lot of what other software distribution package managers already handle and deal with all of the per-platform intricacies. Add on top of this, I assume cargo is aiming to hit 1.0 and maintain compatibility and that could be tricky to define and maintain for all of the different platforms or for even what platforms to support.

I feel like attempting what you are describing is ambitious enough that it would at least start life as a research project that is a cargo plugin. Instead of just having this one issue, I wonder if we should address two things (1) a cross-platform cargo-integrated software distribution method and (2) narrow, concrete use cases for cargo install that we should work to improve.

Additionally, I personally feel like the cargo team or this WG taking on this research project would take so many resources for a feature of limited scope (Rust developers) from more actionable, directly beneficial projects, that this is best done by individuals who are interested in this topic.

it's the only one that we control

Is there something unique to Rust that having control over the packaging scheme benefits us? I'm wondering what advantage this provides us.

[XAMPPRocky]

@epage I think you've misunderstood what I'm proposing. Cargo wouldn't distribute binaries, cargo would still just build from source and be placed in the ~/.cargo/bin. There's no extra maintainability based on platforms as the platforms are whatever cargo is supported. Cargo isn't being proposed as an independent package manager. It's just like npm.

Just because you skip over npm install -g options it doesn't mean other people do, browser-sync is a app that is only delivered through NPM and has had over 1,335,000 downloads in the past month.

[epage]

I think part of my misunderstanding came from:

like there are no other package managers

That implies a scope to handle all problems including

• app-specific assets
• plugins and data files (e.g. shell completions, man pages)
• licensing notifications
• environment variables

. There's no extra maintainability based on platforms as the platforms are whatever cargo is supported. Cargo isn't being proposed as an independent package manager.

Those I described above can involve some platform-specific criteria.

It's just like npm.

It looks like npm is being held up as an example. Could you document it for this discussion. What requirements does it handle? How does it meet those requirements? That might be a big help in understanding what you meant by your earliest comments so we can get on the same page.

[kbknapp]

Having a separate profile for distribution.

I'm not sure I follow here. I would assume that release == distribution and can't see a reason why someone would set something in a release build, but not want it in a distribution build? I.e. ripgrep includes debug information in release builds (and thus for distribution) by design.

I agree with everything else though.

---

On the more general topic of cargo install to elaborate a little from my last replies in the other issue; I don't mean to sound elitist and like people shouldn't release via cargo install, nor do I believe we shouldn't make cargo install a better experience. I do believe it has a purpose, and some of the bugs/issues I've laid out can and should be fixed.

However, having said all that, my argument is against trying to make cargo install into general purpose package manager. Being a general purpose package manager is monumental undertaking and (IMO) far too involved for cargo which should stick to Rust centric development (again, IMO only).

If you're releasing tools designed for Rust development, then sure, I think cargo install is awesome and should improve to meet that goal (code signing, external binary deps, binary only deps, post build steps, etc.). Some, if not all, of those steps making cargo install better at releasing rust-centric tools also make cargo closer to a more general purpose package manager. This is why I think we need to exercise caution when implementing those things, and acknowledging the scope of cargo install. Requiring people to have x language toolchain to install a general purpose application isn't a great option IMO.

These are just my personal opinions and I'm not in charge of cargo nor cargo install so take this all with a grain of salt :wink:

[BurntSushi]

I'm not sure I follow here. I would assume that release == distribution and can't see a reason why someone would set something in a release build, but not want it in a distribution build? I.e. ripgrep includes debug information in release builds (and thus for distribution) by design.

To add context here, standard ripgrep releases do actually strip the binary of debug symbols. Debug info is enabled in the release profile because many of the things I debug in ripgrep are performance problems, and I therefore need to compile in release mode. Since I'm debugging, I also want debug symbols. There is somewhat of a conflation here in that I'm using "release mode" as "compile with optimizations," but, invariably, that's what release mode is. If there were some additional profile that was "compile with optimizations," then I would use that and probably disable debug symbols in the actual "release" profile. If ripgrep had a high incidence of bugs that caused panics (it doesn't), then I might reconsider this and be more aggressive about including debug symbols in released binaries. I could also just tweak the debug = true flag in the release profile whenever I need it, but that's too much of a pain, and I don't care enough about complaints from users of cargo install over a few MB to deal with it.

I don't have many opinions about the wider issue though. Weakly held, it doesn't seem like something I would personally prioritize to be honest. I'd be much more interested in smoothing out the release process for end users that don't care about Rust, because that is by far a much harder task, and maintainers need all the help they can get there.

[kbknapp]

I forgot - when I keep mentioning guides and how-tos for release centric tools and native repos it's because that's step one.

1. Write Guides / How-To's in order to teach people what right looks like
2. Build automation around principals learned from step 1
3. Win

If we start with step 2, only a select few may have the knoweldge to contribute. If we start by teaching, the world of open source opens up. I say this because I don't personally know how to release a deb or rpm or to homebrew or snap or flatpak or AppImage...could I find the info? Sure. But not without struggling through best practices and a bunch of other work that I'm sure there are Rustaceans out there who already have that knowledge! So why duplicate the effort? If there were some basic to intermediate guides on how to release properly for those targets, perhaps I could help build some automation around them, or contribute to some automation at the very least.

Again, just my methodologies behind my thoughts :smile:

[artem-zinnatullin]

Most of applications written in Rust don't require VM/interpreter/etc to work, right. It is a system programming language.

Thus said we shouldn't require users to install, maintain and learn Rust's build/packaging toolchain to consume applications written in Rust.

There are hundreds of programming languages, imagine if every one of them would require you to have own build tools and package managers just to install an app 😸

---

Tbh it's unfortunate that Cargo and Crates.io support not only library distribution/consumption but also executables.

I guess since it does support executables atm, we need to answer "Should it continue supporting distribution/consumption of executables or not?" first as we seem to have different opinions on allocating resources for improving Cargo/Crates support for executables.

And then depending on the answer either implement most of the improvements suggested in this issue or deprecate that support and focus on providing better support for system package managers.

Potential negative side-effect of improving Cargo/Crates support for executables I see is that Rust developers will tend to rely on that thus spreading the feel of complicated/non-standard setup required to install and update applications written in Rust even though technically there are no reasons for that.

[XAMPPRocky]

@artem-zinnatullin Cargo can't ever walk back allowing binary distribution too many crates depend on that being available.

Why do you think this is being proposed as a replacement to traditional distribution. This hasn't ever been said in this thread and in fact I have clarified multiple times that, this is intended as an improvement for Rust developers. Rust developers have cargo, being able to easily cross platform install tools and apps is a great convenience to them. It allows developers to easily get early feedback from other Rust developers.

[epage]

in fact I have clarified multiple times that, this is intended as an improvement for Rust developers

This gets back to my more clearly defining scope. While targeting this at Rust developers might have been the intention, it didn't come across that way for me until this comment.

I think one area people might be confused is:

cargo-install like there are no other package managers as it's the only one that we control and is the package manager a lot of rust developers become familiar with when they are building their app

Looking back at it now, I wonder if you intended this to mean they are familiar with using cargo and so would be comfortable getting apps through it. I had read it as them being familiar with packaging with cargo and could distribute apps through it.

[XAMPPRocky]

Okay, I've gone back and rewritten the original post to give a much clearer sense of direction and hopefully clear up any ambiguity in the text.