Feature: install from manifest

passcod commented 2 years ago

Make a new manifest that specifies a list of crates/tools to be installed. The intent is both to keep a system/user-wide list of tools with versions for local use, and also a list of project-specific tools in a format that can be checked into version control. There should also be a lockfile, for the same mechanism as Cargo deps: version requirements in the manifest, and exact versions in the lockfile.

The lockfile could also include URLs and checksums, providing a measure of security even without signing.
It would be nice to have a set of tools like cargo add, cargo rm to manipulate the manifest.
Needing to manipulate the lockfile points to a wider set of commands than the simple cargo binstall interface can accommodate.
We could embed the manifest in the Cargo.toml, or have our own file. We'll definitely need our own file as a lockfile, so it could make more sense to have our own manifest file.
Out of scope: non-crate tools. E.g. installing valgrind is not our concern.

I think this would be highly valued, and generally be very useful, not just for cargo projects.

I propose ~~(and am working on)~~ the following:

A new tool under this project umbrella, cargo tools (a cargo-tools crate exists, but isn't used).
A new manifest file, Tools.kdl, and its NLJSON lockfile, Tools.lock
Default command cargo tools would print the status (what's installed, what isn't, what can be updated, what can be removed) and an interactive prompt to do something about it (choices: install as lockfile, update compatible, update breaking, do nothing). That way 90% of the time all you want to do is run cargo tools.
A suite of subcommands: cargo tools add, cargo tools rm, cargo tools install, cargo tools update, cargo tools init, cargo tools status, etc

Manifest:

.version 1 // not required

.defaults {
    // Where to install binaries, defaults to `./bin` for projects and to `$HOME/.cargo/bin` for global
    install-into "./vendor"
}

// with version requirement
cargo-binstall version="0.9.1"

// when just any version will do
cargo-watch

// override defaults for this particular tool
watchexec {
    install-into "./bin"
}

A typical manifest could look like:

cargo-binstall
cargo-watch
cargo-make
cargo-edit
cargo-deb version="1.39.1"
cargo-generate-rpm version="0.8.0"

Sample of lockfile (pretty-printed for readability, each top level object would be its own line):

{"version":1}
{"req":{
  "name":"cargo-binstall",
  "version":"0.9.1",
  "metadata":"crates.io",
  "source":"repository",
  "template":{"pkg-fmt":"..."},
  "install-into":"./vendor"
},"pkg":[{
  "version":"0.9.1",
  "target":"x86_64-unknown-linux-musl",
  "source":"https://github.com/ryankurte/cargo-binstall/releases/v0.9.1/download/cargo-binstall-x86_64-unknown-linux-musl.tgz",
  "checksums":{"b3":"c5cd314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"},
  "bins":["cargo-binstall"]
}]}

It's not practical to look for and download all available targets, so only the ones that are installed by the host / runner are written to the lockfile. When a user on a different host target runs cargo install, new files are downloaded and added to the lockfile; this prints a warning/notice to the console. If running with --locked, this causes an error instead. --keep-version is the middle ground, allowing versions to be kept exactly the same as locked, but new files to be downloaded if their target is not in the lockfile already.

NobodyXu commented 2 years ago

Perhaps we can add that as a subcommand and make cargo-binstall a multi-call binary?

passcod commented 2 years ago

Are you concerned about size? I was just thinking of having two binaries.

NobodyXu commented 2 years ago

Are you concerned about size?

Yes.

I don't think we can simply invoke cargo-binstall in cargo-tools since we need to check for the latest release.

passcod commented 2 years ago

I don't mind a multi-call, but on windows we'll still need to distribute two (identical) binaries, I think. Probably the best we can do without going down the path of shipping a dynlib along. Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall pathway?

NobodyXu commented 2 years ago

Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall pathway?

It seems that the user would have to create the symlink themselves in these cases.

cargo-install does not support post installation script.

passcod commented 2 years ago

Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install still works.

NobodyXu commented 2 years ago

Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install still works.

I think we can create bins/cargo-tools.rs and bins/cargo-binstall.rs which simply call the multicall main.

Since the binary is named cargo-tools, multicall would resolve to call the right "main". Same for cargo-binstall.

NobodyXu commented 2 years ago

From https://github.com/ryankurte/cargo-binstall/pull/222#discussion_r927369344:

@passcod Found it!

common_for_install_and_uninstall defines all the metadata for this.

Though I think it might be a good idea to create a format for our own, since the format that make sense to cargo-install but not cargo-binstall, e.g. features, profile, rustc, etc.

It is also not appendable, I will like to format to be appendable for installation though the cargo-tools can remove the duplicate entries when updating installed crates.

Co-op with cargo-install means we have to keep track of the upstream, otherwise we might accidentally break it

passcod commented 2 years ago

We still want to cooperate with the standard files for global installs.

Not entirely sure what to do with them, though, regarding file locking and proper handling and such. I don't really want to embed the cargo lib crate... but maybe that's the lesser evil...

NobodyXu commented 2 years ago

We still want to cooperate with the standard files for global installs.

Other than using cargo install-update -a, I can't think of any other scenario. Also, since users explicitly request the crate to be installed using pre-built binary, they might not want cargo install-update -a to compile them from source.

Not entirely sure what to do with them, though, regarding file locking and proper handling and such.

I'm doing file locking, but it would take a lot of time to get other part right.

I don't really want to embed the cargo lib crate... but maybe that's the lesser evil...

Neither do I, since the CI already takes >5m, adding cargo lib would at least add 1m or maybe 3-4m.

I watched jonhoo's youtube video on trying to speedup compilation when having cargo as a dependency, the conclusion is that there is not much to do except for waiting for the upstream to add more feature flags and perform more optimization.

The problem seems to be mostly about toml_edit taking too long, but I remembered cargo itself also took quite some time, so I really don't want it to further slow down the CI.

passcod commented 2 years ago

The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.

passcod commented 2 years ago

Hmm, toml_edit taking a while is annoying, as I wanted to use that for our own manifest, at least the human-writable bit

NobodyXu commented 2 years ago

The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.

As a user, I really don't want cargo install-update -a to update binaries installed via cargo-binstall. Since I have a mixed set of binary/compiled crates installed locally, I would like two systems to be handled separately.

NobodyXu commented 2 years ago

Hmm, toml_edit taking a while is annoying, as I wanted to use that for our own manifest, at least the human-writable bit

cargo b --timings on M1 (Macbook Air 2020):

cargo b --release --timings with profile.release set to be the same as cargo-binstall:

Since the CI is usually overloaded and I think we only have two cores, IMO it is going to be a lot slower than my M1, taking at least twice or 3x time to build.

Looks like I was wrong about toml_edit, it is taking a lot of time, but its cost relatively small and acceptable compared to cargo.

Though I still don't want to add toml_edit as our CI is already quite slow.

NobodyXu commented 2 years ago

The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.

@passcod IMHO this co-op might not work since cargo-binstall cannot provide information.

InstallInfo contains fields that it is unknown to cargo-binstall, such as features, all_features, no_default_features and rustc (currently unused though).

Even profile is unknown to cargo-binstall though assuming release is reasonable.

Thus I think cargo-install's metafiles are designed to serve cargo-install only and not suitable for cargo-binstall.

passcod commented 2 years ago

Hmm, we could compromise by writing only the simpler toml/v1 file which doesn't have any of these things, so that cargo-update can be used to update via cargo-install if the user so desires but not via us. Drop the json/v2 support and eventually have our own manifest/lockfile that can capture the nuances we care about and work with our own tooling, though we should make it nice enough that other stuff can consume/write it too.

NobodyXu commented 2 years ago

I propose that we can have a manifest like this:

{
    "name": "cargo-binstall",
    "version_req": none,
    "current_version": "0.10.0"
    "source": {
        "type": "Registry",
        "url": "https://crates.io",
     },
     "target": none,
     "bins": [
         "cargo-binstall",
      ],
}
{
    "name": "..."
}

Before installing a crate, we can perform an optional check to see if it is already present. After installation, we can simply append to the file.

When checking for upgrade, we would clean up the file by removing entries with duplicate "name", keeping only the latest one. Then we check each of them one by one for upgrade.

passcod commented 2 years ago

So, the reason I had both tool and package sections in the OP is because there's two concepts of "source": the metadata source, and where the particular file came from (and what target it was installed for etc). I'm not particularly attached to format (toml v json v whatever), but I think that distinction is important, and recording both types of info is useful.

Having the split also meant that a single metafile could be e.g. saved to a repo like the Cargo.lock is, and be used across different hosts, e.g. mac and linux, without conflicting every time it's used on alternating hosts.

Additionally it makes it possible to install from metafile without querying the registry at all, as long as all the required targets are already saved.

NobodyXu commented 2 years ago

Sounds reasonable, I will update the PR after I fixed the error.

NobodyXu commented 2 years ago

@passcod Does something like this sound good to you?

{
    "name": "cargo-binstall",
    "version_req": "*",
    "source": {
        "type": "Registry",
        "url": "https://crates.io",
     },
    "packages": {
         "current_version": "0.10.0",
         "target": "x86_64-unkown-linux-musl",
         "bins": [
             "cargo-binstall",
          ],
      },
}
{
    "name": "..."
}

Having the split also meant that a single metafile could be e.g. saved to a repo like the Cargo.lock is, and be used across different hosts, e.g. mac and linux, without conflicting every time it's used on alternating hosts.

Not sure that is doable, two different hosts can easily have conflict with each other. E.g. they might have different version_req, one might be * and other might be >0.11.1.

The install-into can also be different.

The [[package]] in your spec is where I really don't understand.

If we want to share that single lockfile across different OSes, then I presume that we must have multiple versions of the same package in [[package]].

[[package]]
tool = "cargo-binstall"
version = "0.9.1"
target = "x86_64-unknown-linux-musl"
source = "https://github.com/ryankurte/cargo-binstall/releases/latest/download/cargo-binstall-x86_64-unknown-linux-musl.tgz"
format = "tar+gzip"
checksum = "b3:c5cd314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"
files = ["cargo-binstall"]

[[package]]
tool = "cargo-binstall"
version = "0.9.1"
target = "aarch64-apple-darwin"
source = "https://github.com/ryankurte/cargo-binstall/releases/latest/download/cargo-binstall-aarch64-apple-darwin.zip"
format = "zip"
checksum = "b3:qwed314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"
files = ["cargo-binstall"]

But then how do we differentiate between two nearly identical records for different OSes? Using target seems like a very fragile way to differentiate between them.

This also won't work if you are sharing the lockfile across multiple linux installations. It probably won't even work if you use it in WSL and a native linux installation.

I would rather propose, that the user should share the manifest instead:

version = 1

[defaults.'cfg(target = "x86_64-unkown-linux-musl")']
# Where to install binaries, defaults to `./bin` for projects and to `$HOME/.cargo/bin` for global
install-into = "./vendor"

[tools]
cargo-binstall = "0.9.1"
cargo-watch = "8.1.1"
watchexec = "1.19.0"

This is much easier to make portable and you could even put it in a portable HDD/SDD.

Additionally it makes it possible to install from metafile without querying the registry at all, as long as all the required targets are already saved.

I don't think this is a good idea, we should have a separate file for this and it can only be a best effort since the upstream can change this at anytime.

passcod commented 2 years ago

well, the original thing is to have both the manifest and the lockfile in tandem, modelled like Cargo. Cargo.lock has a fully resolved tree, so if you dev on linux and CI runs on windows, the lockfile need not change, and similarly across targets. however a lockfile is useless on its own, it's only paired with the manifest that it makes sense.

similarly here, the op design is for these two files together. however we can't really reasonably resolve the entire thing so that's why there's provisions for updating the lockfile with new targets even when doing a "non updating" install. the design is also specifically for a project-local manifest and lockfile, with the global use a special case, rather than being made for the global use first. there's surely edge cases and flaws in this first draft as you point out that would need to be worked out but that's the intent.

now, I think perhaps what would be better at this point is we can define something completely different for our own .crates.toml-like global metafile with separate considerations, something more like your original. I'd want to then:

split it from this issue to avoid derailing further ;P
define what the goals for the metafile are. what problems does it solve, what the use will be now and in future
explore a bit the interaction between the metafile and the theoretical manifest+lock described here. are they complementary? conflicting?
consider the external use. if an auxillary project like cargo-update wants to use it, could they? if a project like quickinstall, which has its own cli, wants to use it, could they? is that something we want to enable or to discourage?

passcod commented 2 years ago

I also kinda want to avoid the proliferation of many different files that we write to. ...or maybe we say, let's go and write as many files as we want to, but I'd say let's make our own config folder for all this. there's other stuff we might want to start doing, like keeping caches e.g. for hsts. and having an actual config file, I think that came up once or twice recently

NobodyXu commented 2 years ago

now, I think perhaps what would be better at this point is we can define something completely different for our own .crates.toml-like global metafile with separate considerations, something more like your original.

Since we have agreed to use a different format for global installations, let's keep the discussion in #252 !

NobodyXu commented 2 years ago

Now that we have the global $CARGO_HOME/binstall/crates-v1.json, I think it is time to support upgrading installed bins.

I propose something like:

cargo binstall --upgrade

to upgrade all installed binaries and

cargo binstall --uprade $crate1 $crate2:$ver2

to upgrade specific binaries only and also enable them to provide a specification for upgrading (it cannot be used to downgrade though).

NobodyXu commented 2 years ago

Now with #270 and cargo-update adding support for cargo-binstall, perhaps we don't need --upgrade anymore?

ryankurte commented 2 years ago

ooh, i like the idea of this! would it be too self important to call this Tools.toml instead of CargoTools.toml?

ryankurte commented 1 year ago

hey thinking about this a bit more, how do you manage multiple projects with different tool requirements? it feels like a global manifest is a different problem to a per-project manifest (whether that's at a crate or workspace level), particularly given projects could have different tool version requirements.

if you want a -project- to be able to specify required tools it might be plausible to add a tag in Cargo.toml then use the existing build-dependencies and lockfile... as well as necessary to locate the binaries -within- the project / get that on the path somehow?

passcod commented 1 year ago

Yeah, the way I was thinking is that in project mode, we install to a directory inside the project, so it's completely separate from the global installs. There's no convention in rust to have a project-local bin folder that the tooling uses like in Node (node_modules/.bin), which makes this a bit unfortunate to use (vendor/tool args rather than tool args, for example), but plenty of projects have a "scripts" folder, which this would just be an extension of, I guess.

passcod commented 1 year ago

There's also the cargo xtask pattern, but I'm not sure if we could reasonably use that or hook into it for anything.

passcod commented 9 months ago

Implemented by (third party project) cargo-run-bin, see #1514.

cargo-bins / cargo-binstall

Feature: install from manifest #176