Closed passcod closed 9 months ago
Perhaps we can add that as a subcommand and make cargo-binstall
a multi-call binary?
Are you concerned about size? I was just thinking of having two binaries.
Are you concerned about size?
Yes.
I don't think we can simply invoke cargo-binstall
in cargo-tools
since we need to check for the latest release.
I don't mind a multi-call, but on windows we'll still need to distribute two (identical) binaries, I think. Probably the best we can do without going down the path of shipping a dynlib along. Also not entirely sure how multicall binaries work in the cargo context, especially with the cargo install cargo-binstall
pathway?
Also not entirely sure how multicall binaries work in the cargo context, especially with the
cargo install cargo-binstall
pathway?
It seems that the user would have to create the symlink themselves in these cases.
cargo-install
does not support post installation script.
Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and cargo install
still works.
Right, well, that's not a great experience. I wonder if we could rig it such that it defaults to two binaries, but produces a multicall given a feature. That way we can generate prebuilds with the smaller multicall setup, and
cargo install
still works.
I think we can create bins/cargo-tools.rs and bins/cargo-binstall.rs which simply call the multicall main.
Since the binary is named cargo-tools, multicall would resolve to call the right "main". Same for cargo-binstall.
From https://github.com/ryankurte/cargo-binstall/pull/222#discussion_r927369344:
@passcod Found it!
common_for_install_and_uninstall defines all the metadata for this.
Though I think it might be a good idea to create a format for our own, since the format that make sense to cargo-install but not cargo-binstall, e.g. features, profile, rustc, etc.
It is also not appendable, I will like to format to be appendable for installation though the cargo-tools can remove the duplicate entries when updating installed crates.
Co-op with cargo-install means we have to keep track of the upstream, otherwise we might accidentally break it
We still want to cooperate with the standard files for global installs.
Not entirely sure what to do with them, though, regarding file locking and proper handling and such. I don't really want to embed the cargo lib crate... but maybe that's the lesser evil...
We still want to cooperate with the standard files for global installs.
Other than using cargo install-update -a
, I can't think of any other scenario.
Also, since users explicitly request the crate to be installed using pre-built binary, they might not want cargo install-update -a
to compile them from source.
Not entirely sure what to do with them, though, regarding file locking and proper handling and such.
I'm doing file locking, but it would take a lot of time to get other part right.
I don't really want to embed the cargo lib crate... but maybe that's the lesser evil...
Neither do I, since the CI already takes >5m, adding cargo lib would at least add 1m or maybe 3-4m.
I watched jonhoo's youtube video on trying to speedup compilation when having cargo
as a dependency, the conclusion is that there is not much to do except for waiting for the upstream to add more feature flags and perform more optimization.
The problem seems to be mostly about toml_edit
taking too long, but I remembered cargo
itself also took quite some time, so I really don't want it to further slow down the CI.
The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.
Hmm, toml_edit taking a while is annoying, as I wanted to use that for our own manifest, at least the human-writable bit
The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.
As a user, I really don't want cargo install-update -a
to update binaries installed via cargo-binstall
.
Since I have a mixed set of binary/compiled crates installed locally, I would like two systems to be handled separately.
Hmm, toml_edit taking a while is annoying, as I wanted to use that for our own manifest, at least the human-writable bit
cargo b --timings
on M1 (Macbook Air 2020):
cargo b --release --timings
with profile.release
set to be the same as cargo-binstall
:
Since the CI is usually overloaded and I think we only have two cores, IMO it is going to be a lot slower than my M1, taking at least twice or 3x time to build.
Looks like I was wrong about toml_edit
, it is taking a lot of time, but its cost relatively small and acceptable compared to cargo
.
Though I still don't want to add toml_edit
as our CI is already quite slow.
The idea for meta files support, separate from this issue, was for cargo-update to be modified to pull from cargo-binstall when available, originally.
@passcod IMHO this co-op might not work since cargo-binstall
cannot provide information.
InstallInfo
contains fields that it is unknown to cargo-binstall
, such as features
, all_features
, no_default_features
and rustc
(currently unused though).
Even profile
is unknown to cargo-binstall
though assuming release
is reasonable.
Thus I think cargo-install
's metafiles are designed to serve cargo-install
only and not suitable for cargo-binstall
.
Hmm, we could compromise by writing only the simpler toml/v1 file which doesn't have any of these things, so that cargo-update can be used to update via cargo-install if the user so desires but not via us. Drop the json/v2 support and eventually have our own manifest/lockfile that can capture the nuances we care about and work with our own tooling, though we should make it nice enough that other stuff can consume/write it too.
I propose that we can have a manifest like this:
{
"name": "cargo-binstall",
"version_req": none,
"current_version": "0.10.0"
"source": {
"type": "Registry",
"url": "https://crates.io",
},
"target": none,
"bins": [
"cargo-binstall",
],
}
{
"name": "..."
}
Before installing a crate, we can perform an optional check to see if it is already present. After installation, we can simply append to the file.
When checking for upgrade, we would clean up the file by removing entries with duplicate "name"
, keeping only the latest one.
Then we check each of them one by one for upgrade.
So, the reason I had both tool
and package
sections in the OP is because there's two concepts of "source": the metadata source, and where the particular file came from (and what target it was installed for etc). I'm not particularly attached to format (toml v json v whatever), but I think that distinction is important, and recording both types of info is useful.
Having the split also meant that a single metafile could be e.g. saved to a repo like the Cargo.lock is, and be used across different hosts, e.g. mac and linux, without conflicting every time it's used on alternating hosts.
Additionally it makes it possible to install from metafile without querying the registry at all, as long as all the required targets are already saved.
Sounds reasonable, I will update the PR after I fixed the error.
@passcod Does something like this sound good to you?
{
"name": "cargo-binstall",
"version_req": "*",
"source": {
"type": "Registry",
"url": "https://crates.io",
},
"packages": {
"current_version": "0.10.0",
"target": "x86_64-unkown-linux-musl",
"bins": [
"cargo-binstall",
],
},
}
{
"name": "..."
}
Having the split also meant that a single metafile could be e.g. saved to a repo like the Cargo.lock is, and be used across different hosts, e.g. mac and linux, without conflicting every time it's used on alternating hosts.
Not sure that is doable, two different hosts can easily have conflict with each other.
E.g. they might have different version_req
, one might be *
and other might be >0.11.1
.
The install-into
can also be different.
The [[package]]
in your spec is where I really don't understand.
If we want to share that single lockfile across different OSes, then I presume that we must have multiple versions of the same package in [[package]]
.
[[package]]
tool = "cargo-binstall"
version = "0.9.1"
target = "x86_64-unknown-linux-musl"
source = "https://github.com/ryankurte/cargo-binstall/releases/latest/download/cargo-binstall-x86_64-unknown-linux-musl.tgz"
format = "tar+gzip"
checksum = "b3:c5cd314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"
files = ["cargo-binstall"]
[[package]]
tool = "cargo-binstall"
version = "0.9.1"
target = "aarch64-apple-darwin"
source = "https://github.com/ryankurte/cargo-binstall/releases/latest/download/cargo-binstall-aarch64-apple-darwin.zip"
format = "zip"
checksum = "b3:qwed314cea5aca5391427bf9885d11f1fbc86d79679353a36d1b89480422149d"
files = ["cargo-binstall"]
But then how do we differentiate between two nearly identical records for different OSes?
Using target
seems like a very fragile way to differentiate between them.
This also won't work if you are sharing the lockfile across multiple linux installations. It probably won't even work if you use it in WSL and a native linux installation.
I would rather propose, that the user should share the manifest instead:
version = 1
[defaults.'cfg(target = "x86_64-unkown-linux-musl")']
# Where to install binaries, defaults to `./bin` for projects and to `$HOME/.cargo/bin` for global
install-into = "./vendor"
[tools]
cargo-binstall = "0.9.1"
cargo-watch = "8.1.1"
watchexec = "1.19.0"
This is much easier to make portable and you could even put it in a portable HDD/SDD.
Additionally it makes it possible to install from metafile without querying the registry at all, as long as all the required targets are already saved.
I don't think this is a good idea, we should have a separate file for this and it can only be a best effort since the upstream can change this at anytime.
well, the original thing is to have both the manifest and the lockfile in tandem, modelled like Cargo. Cargo.lock has a fully resolved tree, so if you dev on linux and CI runs on windows, the lockfile need not change, and similarly across targets. however a lockfile is useless on its own, it's only paired with the manifest that it makes sense.
similarly here, the op design is for these two files together. however we can't really reasonably resolve the entire thing so that's why there's provisions for updating the lockfile with new targets even when doing a "non updating" install. the design is also specifically for a project-local manifest and lockfile, with the global use a special case, rather than being made for the global use first. there's surely edge cases and flaws in this first draft as you point out that would need to be worked out but that's the intent.
now, I think perhaps what would be better at this point is we can define something completely different for our own .crates.toml-like global metafile with separate considerations, something more like your original. I'd want to then:
I also kinda want to avoid the proliferation of many different files that we write to. ...or maybe we say, let's go and write as many files as we want to, but I'd say let's make our own config folder for all this. there's other stuff we might want to start doing, like keeping caches e.g. for hsts. and having an actual config file, I think that came up once or twice recently
now, I think perhaps what would be better at this point is we can define something completely different for our own .crates.toml-like global metafile with separate considerations, something more like your original.
Since we have agreed to use a different format for global installations, let's keep the discussion in #252 !
Now that we have the global $CARGO_HOME/binstall/crates-v1.json
, I think it is time to support upgrading installed bins.
I propose something like:
cargo binstall --upgrade
to upgrade all installed binaries and
cargo binstall --uprade $crate1 $crate2:$ver2
to upgrade specific binaries only and also enable them to provide a specification for upgrading (it cannot be used to downgrade though).
Now with #270 and cargo-update
adding support for cargo-binstall
, perhaps we don't need --upgrade
anymore?
ooh, i like the idea of this! would it be too self important to call this Tools.toml
instead of CargoTools.toml
?
hey thinking about this a bit more, how do you manage multiple projects with different tool requirements? it feels like a global manifest is a different problem to a per-project manifest (whether that's at a crate or workspace level), particularly given projects could have different tool version requirements.
if you want a -project- to be able to specify required tools it might be plausible to add a tag in Cargo.toml
then use the existing build-dependencies
and lockfile... as well as necessary to locate the binaries -within- the project / get that on the path somehow?
Yeah, the way I was thinking is that in project mode, we install to a directory inside the project, so it's completely separate from the global installs. There's no convention in rust to have a project-local bin folder that the tooling uses like in Node (node_modules/.bin
), which makes this a bit unfortunate to use (vendor/tool args
rather than tool args
, for example), but plenty of projects have a "scripts" folder, which this would just be an extension of, I guess.
There's also the cargo xtask pattern, but I'm not sure if we could reasonably use that or hook into it for anything.
Implemented by (third party project) cargo-run-bin, see #1514.
Make a new manifest that specifies a list of crates/tools to be installed. The intent is both to keep a system/user-wide list of tools with versions for local use, and also a list of project-specific tools in a format that can be checked into version control. There should also be a lockfile, for the same mechanism as Cargo deps: version requirements in the manifest, and exact versions in the lockfile.
I think this would be highly valued, and generally be very useful, not just for cargo projects.
I propose
(and am working on)the following:cargo tools
(acargo-tools
crate exists, but isn't used).Tools.kdl
, and its NLJSON lockfile,Tools.lock
cargo tools
would print the status (what's installed, what isn't, what can be updated, what can be removed) and an interactive prompt to do something about it (choices: install as lockfile, update compatible, update breaking, do nothing). That way 90% of the time all you want to do is runcargo tools
.cargo tools add
,cargo tools rm
,cargo tools install
,cargo tools update
,cargo tools init
,cargo tools status
, etcManifest:
A typical manifest could look like:
Sample of lockfile (pretty-printed for readability, each top level object would be its own line):
It's not practical to look for and download all available targets, so only the ones that are installed by the host / runner are written to the lockfile. When a user on a different host target runs
cargo install
, new files are downloaded and added to the lockfile; this prints a warning/notice to the console. If running with--locked
, this causes an error instead.--keep-version
is the middle ground, allowing versions to be kept exactly the same as locked, but new files to be downloaded if their target is not in the lockfile already.