search

cargo-bins / cargo-binstall

Binary installation for rust projects
GNU General Public License v3.0
1.4k stars 50 forks source link

Feedback / Experience / Adventures meta-issue #19

Open ryankurte opened 3 years ago

ryankurte commented 3 years ago

A meta-issue for the experience of using this, if you have any thoughts / feelings / ideas please comment here!

fasterthanlime commented 2 years ago

Discovered cargo-binstall through cargo-watch's readme, instantly worked, no complaints.

I've been trying to run cargo binstall some-random-package, and it's often failed due to some-random-package not having the right "releases layout" / not having binaries at all / not having any cargo-binstall config. I'm wondering if cargo-binstall should fall back to building from source in these cases (maybe after asking / only if a CLI flag is passed?). I'd love for cargo-binstall to be my one-stop-shop for installing Rust utilities. If there's a binary, yay! If not, I'll take the compile time hit.

ryankurte commented 2 years ago

Discovered cargo-binstall through cargo-watch's readme, instantly worked, no complaints.

glad to hear it ^_^

I've been trying to run cargo binstall some-random-package, and it's often failed due to some-random-package not having the right "releases layout" / not having binaries at all / not having any cargo-binstall config.

ahh, yeah, i'm definitely open to improvements if you have ideas! have contemplated making the discovery process a bit more robust/overzealous if there's no config, or some kind of unofficial registry so where folks have CI builds we could link to their binaries without Cargo.toml config, but i think actually building them would move this from (hopefully) helpful tool to accidentally a package manager.

I'm wondering if cargo-binstall should fall back to building from source in these cases (maybe after asking / only if a CLI flag is passed?). I'd love for cargo-binstall to be my one-stop-shop for installing Rust utilities. If there's a binary, yay! If not, I'll take the compile time hit.

sounds like a great idea to me! given it's already interactive a prompt w/ ci flag to automagically do it would probably make sense?

fasterthanlime commented 2 years ago

given it's already interactive a prompt w/ ci flag to automagically do it would probably make sense?

That sounds great! I vaguely remembered seeing some interactive prompts already in cargo-binstall hence my suggestion.

pinage404 commented 2 years ago

It seems that both terms are used in a confusing way :

pinage404 commented 2 years ago

It's hard to understand why format :

Does the format refer to the same concept ?

ryankurte commented 2 years ago

thanks for the reports!

wrt. bin-dir and bin-path, this appears to be a typo / should all be called bin-dir

wrt. format, those are actually two (unfortunately named) different concepts, the first refers to the archive format (eg. .tgz), the second to the binary format (which needs a .exe appended for windows).

it would be nice to rename these to highlight the difference, and it might make sense to include the . in both cases, but i would aim to avoid any breaking changes. it's probably worth looking at whether there's a compatible way to mitigate these and definitely able to improve the documentation to clarify both!

pinage404 commented 2 years ago

for the format, this could be done without breaking changes, supporting both the deprecated format and the new archive-format and binary-format

auronandace commented 2 years ago

Would it be possible to add a scan feature to see what you have installed via cargo install and list whether they are also able to be installed via cargo binstall? Sort of a binstall version of this:

https://github.com/nabijaczleweli/cargo-update

somehowchris commented 2 years ago

Something like "install from Cargo.toml" would be nice. We have a case where some people joined the team and didn't know the tools and just listing them one by one can be a long process on multiple machines.

Also this would give us the oportinity to profit from tooling like dependabot, renovate, synk,... which would give you a heads up if some tools have been updated, so you should update your tooling too (or the Cargo.toml which you then can just i.e. cargo binstall --manifest .). This could be included in things like git hooks or make files so all devs know they are running the same thing.

Also, tried to add cicd with cross and a lot more to just have binstall support. I think this is a bit hurdle for some which then just get frustrated. But I'm trying to get a github action running for something like this. (would also be lovely to have #1 with that 😄 )

But in general, just awesome, saves a lot of time

passcod commented 2 years ago

Oh, that's interesting! Are you looking for a list of crates or also something like having version requirements or a lockfile?

somehowchris commented 2 years ago

Yes, well would be nice.

Just has been a small pain point for me today. Me personally, I have a small repo with a Cargo.toml which is watched by renovate, so I know when a tool has been updated. For a) update the container image and b) actualy know what changed.

So lets say if v0.1.1 was released and I have v0.1.0 on my laptop, it would be lovely to just have something like cargo binstall update or as said cargo binstall --manifest . to update the binaries. (or as mentioned have them as githooks for something like git pull)

Edit: Also just had a thought about rust equivalent tooling to like npms husky, and tooling used alongside cargo like cargo-watch or wasm-bindgen-cli, specified in the Cargo.toml under something like package.metadata.binstall-tools to not get confused with other dependencies and to have them versioned per project as they are needed for it. This would not be included in usual tooling like renovate or dependabot but I know renovate can easily be configured to include these and they would be for sure be open to have a PR to include this (behind the scenes it would use cargos registries & functionality as usual)

somehowchris commented 2 years ago

Just saw that there is some usage report thing going on for cargo-quickinstall. Mind if I ask, why aren't you tracking too?

Over at cargo-all-features I know that there are about 8k recent downloads via crates.io but would there be interest in binaries (aka to be able to use cargo-binstall)? We don't know as we do not have that kind of inisght

passcod commented 2 years ago

The usage report for quickinstall is because:

I don't personally want to collect tracking information; Ryan may have a different opinion. It's a service to keep up, potentially bills to pay, security to be looked after; there's privacy considerations. It's a very different ballgame to just providing a tool.

Additionally, I don't see the interest in maintaining two tracking services. The features you propose could very well be built into quickinstall's solution; we even have a different user agent so provenance can in theory be differentiated.

Also note that binstall pulls directly from the urls, it doesn't transit or cache, so if you do have binaries up you can get usage stats from those. For github release artifacts, that information is exposed in the github UI (and API).

ryankurte commented 2 years ago

yep we're on the same page @passcod, if we can feed better user-agent information (or anything else) to the APIs we're interacting with that could be neat, but definitely prefer no tracking here.

(thinking about feedback, we could have a if this doesn't work for you, please open an issue with the following information link in the help or something to help point people back here when things go wrong?)

somehowchris commented 2 years ago

So, during the last weekend I made some PRs, some intentionally (more or less planed) and some more spontaneousas I ran into some small issues or saw some potential. You @ryankurte engeuraged me to open issues about some things. Just to reiterate, I like this project, so I would like to contribute. Its up to you what you accept, ignore or reject. I'm motivated by learning and also by hearing/reading others opinions.

In order to proceed, I listed some of my wishes/motivations below, I would love to hear from you on which you would like to see PRs, issues or general discussions, or plainly reject. These may include some of the things already merged but may paint the bigger picture

Some ideas:

More spontaneous ones:

If you feel like discussions or comment threads could be too big, feel free to email me

passcod commented 2 years ago

It's great to see enthusiasm for this project! It's a big list and there's certainly a lot of potential. I think primarily what we're looking at moderating here is going too fast and not justifying complexity which will come back to bite us in the long run. One thing in particular you've touched on is this:

If cargo-binstall is being used across hundreds or thousands of ci-pipelines

That would be great success! It would also mean that a breaking change will break thousands of uses. I think we want to avoid that, and a good way to avoid this is to be careful on what we implement now so we are less likely to revisit it later. We can move a tad faster than Rust itself ;) but let's not go too far in the "move fast and break things" direction either.

One thing I see often in open source contributions vs open source maintainership is this idea that Ryan touched on a bit earlier: we want to fix problems that exist. But it's not enough that you have the issue. As maintainers, we're looking at the software as it is used by a lot of people. You having the issue is one person. Sometimes it's obvious that something will affect more people. Sometimes it's much less so. Other times, an issue might be better solved outside of this project, and "fixing" it here is a false herring.

We also want to consider the direction for the project. As a secondary maintainer, this is something that I'll often defer to Ryan about, as it's his project in the first place, right? It's things like, do we want to be a one-stop-shop where you hit cargo binstall package and it figures out the right thing to do, even if it has a bunch of "magic" (yes), and do we want to track users (no). Going forward, it might be do we want to focus on the human experience and discourage CI use or do we want to make both ways as ergonomic as possible or do we want to focus on automated use and near-automated use and perhaps accept some minor annoyances for human use. How much of a "drop in replacement for cargo install" do we want to be? Is that even a goal? That kind of thing.

So, let's take a step back, let's take a breath, let's talk some things out and see where we fall. I think also something to keep in mind is both Ryan and I are fairly busy and don't have a lot of time to dedicate to this project: so it's both great to see progress being made, and a bit overwhelming at the same time.

Over at diesel there is an issue to support binary installs

Can you expand more on this? I can't see how that would fit/work with what diesel is.

NobodyXu commented 2 years ago

I am working on adding cargo-binstall to taiki-e's install-action https://github.com/taiki-e/install-action/pull/8/ so that github action can use cargo-binstall.

passcod commented 2 years ago

I'm working on the "installing a suite of tools from manifest" thing btw

yonkeltron commented 1 year ago

I found out about cargo-binstall through this awesome introductory blog post about cargo-dist. I am admittedly late to the party. Unfortunately, when I went to try this out at work, the corporate proxy got in the way and I can't yet figure out how to specify a certificate bundle like I can with other apps. I already have the customary environment variables set properly such as CARGO_HTTP_CAINFO, SSL_CERT_FILE, and SSL_CERT_PATH. This is a bit frustrating but doesn't have to be the end. Excited to learn more.

kekonn commented 1 year ago

I found out about cargo-binstall through this awesome introductory blog post about cargo-dist. I am admittedly late to the party. Unfortunately, when I went to try this out at work, the corporate proxy got in the way and I can't yet figure out how to specify a certificate bundle like I can with other apps. I already have the customary environment variables set properly such as CARGO_HTTP_CAINFO, SSL_CERT_FILE, and SSL_CERT_PATH. This is a bit frustrating but doesn't have to be the end. Excited to learn more.

Can confirm this solved it for me as well.

echoix commented 1 year ago

It would be nice in the README to have an example of a one-liner to download and install (extract and move) from the links given. At each time I want to use binstall on a new setup without a desktop, I totally forget the syntax when it comes to tar, so I always end up searching how to do it. At least for the linux links

hawkw commented 10 months ago

Binaries installed using cargo-binstall will not execute on NixOS, because they have an interpreter path that points to:

interpreter /lib64/ld-linux-x86-64.so.2

but for locally-built executables, the interpreter path is to the ld in the Nix store:

eliza@noctis:~ $ file /home/eliza/.cargo/bin/cargo-binstall
/home/eliza/.cargo/bin/cargo-binstall: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.10.0, with debug_info, not stripped

note that the interpreter path is:

interpreter /nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/lib/ld-linux-x86-64.so.2

I first encountered this issue while trying to install Oranda using cargo-binstall (https://github.com/axodotdev/oranda/issues/606) , but I'd imagine this is not an Oranda-specific issue and instead occurs whenever using cargo-binstall to install binaries on NixOS.

I'm not sure if there's really anything that cargo-binstall can do to fix this. It's probably better for NixOS users to just install binaries using Nix (I reached for cargo binstall in the hopes that I could install a newer version than what's packaged for Nix). If I'm wrong and there is a way to make it play nice with NixOS, that would be cool, but I imagine the best solution is probably just to document that cargo-binstall doesn't work on NixOS, so that other people don't run into similar problems.

NobodyXu commented 10 months ago

@hawkw You could use --tarfers to specifically use the musl target instead, that will definitely work. We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

hawkw commented 10 months ago

@hawkw You could usw the musl target instead, that will definitely work. We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

Oh, that's a great idea, installing statically-linked musl targets on NixOS would probably work fine.

eitsupi commented 10 months ago

You could use --tarfers to specifically use the musl target instead, that will definitely work. We can modify crates/detect-targets to disable gnu if the interpreter does not exist or is not executable.

Related to this, I recently ran into a problem that I believe was caused by gnu being installed instead of musl. (PRQL/prql#3336) Could you consider changing the default priority to install musl first?

Thanks for wonderful tool!

NobodyXu commented 10 months ago

Could you consider changing the default priority to install musl first?

We could add another glibc version check, if it is lower than a certain threshold then we would fallback to musl instead.

We can also provide an environment variable for overriding targets.

NobodyXu commented 10 months ago

@eitsupi After thinking this again, I think this is more of an issue with mdbook pre-built and it's a good idea to open an issue there.

You could also try using quickinstall build, it uses older glibc.

The best way forward is to check the glibc version used by the binary, if it is newer than the system-installed glibc, then fallback to musl

eitsupi commented 10 months ago

@eitsupi After thinking this again, I think this is more of an issue with mdbook pre-built and it's a good idea to open an issue there.

Thanks for pointing that! I'm sorry I don't know exactly what the problem is, but perhaps it has something to do with the fact that they are building here using old Ubuntu 20.04? https://github.com/rust-lang/mdBook/blob/02f3823e4caae28cc0b1ceac284f81743f8fd29b/.github/workflows/deploy.yml#L26-L31

Edit: I found the issue about that. Thanks! rust-lang/mdBook#1954

In any case, a fallback mechanism to musl would definitely be useful.

Sculas commented 10 months ago

I'm not sure why, but cargo-binstall is really slow for me. I am authenticated to GitHub using gh, so it's not GitHub that's rate-limiting me here. This was me installing cargo-nextest for the first time:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 WARN The package cargo-nextest v0.9.57 will be downloaded from github.com
 INFO This will install the following binaries:
 INFO   - cargo-nextest.exe (cargo-nextest.exe -> C:\Users\$USER\.cargo\bin\cargo-nextest.exe)
 INFO Installing binaries...
 INFO Done in 196.7039375s

Then, I ran the same command again to check how long it took when cargo-nextest was already installed:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 INFO resolve: cargo-nextest v0.9.57 is already installed, use --force to override
 INFO Done in 45.3914203s

Taking a look at other issues in this repo, I'm definitely sure that this shouldn't take that long.

echoix commented 10 months ago

It looks like it is on Windows, can you rule out any antivirus interference in that test?

Sculas commented 10 months ago

It looks like it is on Windows, can you rule out any antivirus interference in that test?

I first ran cargo uninstall cargo-nextest, then I disabled "Real-time protection" in Windows Defender (I don't have any other AV), and then ran cargo binstall cargo-nextest -y again. These are the results:

$ cargo binstall cargo-nextest -y
 INFO resolve: Resolving package: 'cargo-nextest'
 WARN The package cargo-nextest v0.9.57 will be downloaded from github.com
 INFO This will install the following binaries:
 INFO   - cargo-nextest.exe (cargo-nextest.exe -> C:\Users\$USER\.cargo\bin\cargo-nextest.exe)
 INFO Installing binaries...
 INFO Done in 197.1092732s

No dice, unfortunately. I've reran the same commands, but now with debug logging enabled:

$ cargo binstall cargo-nextest -y --log-level debug

Log: contents.log

passcod commented 10 months ago

Can you provide that log again but with --json-output and at trace level so we can see where the time spent is? thanks

Sculas commented 10 months ago

Can you provide that log again but with --json-output and at trace level so we can see where the time spent is? thanks

I've done the same steps as before (uninstall, turn off AV), here's the command I ran:

$ cargo binstall cargo-nextest -y --log-level trace --json-output

Log: contents.log

Looking at the log myself, the 2 things that take the longest are the download from crates.io and the download from GitHub. Why is that? I have a gigabit fiber connection (and a speed test proves that), so it's not my internet being slow. I'm clueless.

Fyko commented 8 months ago

Howdy folks! Really enjoying cargo-binstall. I'm not sure if this already exists (because it's not noted in SUPPORT.md), but it would be lovely to support a bin name that's different from the package name. For a project I'm working on, scyllax-cli, I'd like the bin to be named scyllax, which can easily be done with Cargo.toml's [[bin]] feature, but it doesn't seem to work with binstall.

NobodyXu commented 8 months ago

That's indeed a usecase we yet to support: We assume that the release name contains package name, not binary name.

I'm working on dist-manifest support which might help this, and maybe we can also support this directly in existing fetchers.

Fyko commented 8 months ago

I just discovered, it seems to work for taplo-cli.

$ cargo binstall taplo-cli -y
$ which taplo-cli
taplo-cli not found
$ which taplo
/home/carter/.cargo/bin/taplo
passcod commented 8 months ago

Yes, this should work as we do parse the Cargo.toml. Can you clarify and open an issue with what "doesn't work" in your case?

ilyagr commented 2 weeks ago

Our project just had a mis-adventure with binstall. It seems that cargo binstall preferred to download binaries from QuickInstall as opposed to our own, and they turned out to be broken.

https://github.com/martinvonz/jj/issues/3844#issuecomment-2156103334 https://github.com/cargo-bins/cargo-quickinstall/issues/250

Is there a way to specify in our Cargo.toml that our binaries should be preferred? We already have some cargo binstall config:

https://github.com/martinvonz/jj/blob/65a988e3d2d6b3ed201ed8c2b4952ef08344e55f/cli/Cargo.toml#L118-L121

Another issue is that we only provide musl binaries, so perhaps binstall prefers QuickInstall because it has gnu binaries.

NobodyXu commented 2 weeks ago

Another issue is that we only provide musl binaries, so perhaps binstall prefers QuickInstall because it has gnu binaries

Yes that's why quickinstall is preferred, on gnu target it will try crate-meta-data => quickinstall, if it fails, then fallback to musl.

We can add a new configuration key to Cargo.toml to disable quickinstall fallback, for now you can either

Thanks for reporting this!