search

rust-cli / team

CLI working group
https://rust-cli.github.io/book
MIT License
296 stars 34 forks source link

Packaging and distributing apps #8

Open killercup opened 6 years ago

killercup commented 6 years ago

(moderated summary by @epage)

Context:

Assumptions and Implications

Other solutions

Plan of attack

  1. "How to package" documentation
  2. Automation
    • Implement stager
    • Reach out to binary packaging tools to consolidate effort with stager

In-ecosystem resources

Open Issues

Open Questions

@killercup's original post

In the first meeting, we identified packaging and distributing CLI apps a something we should improve.

We need to have a best-practice solution for getting from "I have a crate with a binary" to "I can ship this app and don't have to worry about cross-platform installation stuff".

Edit 2018-03-19: This issue is about distribution of rust binaries without using cargo-install.

killercup commented 6 years ago
killercup commented 6 years ago

Distribute documentation:

kbknapp commented 6 years ago
killercup commented 6 years ago

From the etherpad:

  • Guides/How-Tos
    • Deb, RPM, AUR, Homebrew, Snap, Flatpak, AppImage, nixpkgs
  • Automation
    • cargo subcommands
    • Using CI

If cargo-install is an important use case for dev tools, how much should we look at improving it?

XAMPPRocky commented 6 years ago

If cargo-install is an important use case for dev tools, how much should we look at improving it?

I think this is part of a lot broader area of "crates.io's treatment of executables.". Which is something that previously has been very vague. Part of distribution should be codifying how crates.io handles and promotes executables or lack thereof.

I'm unsure how we should go about this as there isn't really a process for requesting crates.io features.

jaemk commented 6 years ago

For distributing my cli apps I made https://github.com/jaemk/self_update to support a "self update" command that pulls the latest github release built using https://github.com/japaric/trust/

spacekookie commented 6 years ago

@killercup When it comes to automatically generating debian packages (and there is already cargo-deb for that) I think it's even more important to generate source packages for distributions to include (or to build a PPA with). And that is something that cargo-deb can't yet to.

See cargo-deb#40

matklad commented 6 years ago

For windows specifically, it would be nice to be able to add manifest to the binary: https://github.com/rust-lang/rfcs/issues/721.

sophiajt commented 6 years ago

A couple more thoughts after brainstorming with @killercup:

With CLI apps that are built by doing "cargo install", we have a pretty straightforward path we could go down with regards to building installers for people.

For example, what if we had a new cargo plugin, or separate tool, that could take a "cargo install"-able CLI app and then, using a bit of convention over configuration, build you an installable package for a distro or use a simple Windows installer .exe?

This seems doable because there are really only a couple steps when doing CLI work specifically: "build my binary" and "put it in my path", which distros and installers should be able to handle no problem. @killercup mentioned that we might even want to go a step further and offer some project templates that drop in whatever metadata is required for building .deb, homebrew, etc. From there maybe the cargo plugin can either be configured and/or searches for packaging tools it knows about and builds using them.

These combined with something like https://github.com/japaric/trust/ means folks could take an easy extra step to see what tweaks they'd need to make to get it to work across more platforms.

Screwtapello commented 6 years ago

This seems doable because there are really only a couple steps when doing CLI work specifically: "build my binary" and "put it in my path", which distros and installers should be able to handle no problem.

Eh, that kind of depends on the tool, and how polished an experience you want it to be. Even static binaries might want to include manpages, generic documentation, default config files, command-line completion integration, data files...

I suspect no single tool is going to be able to support many or varied platforms, because platform conventions vary so widely, and because so few tool-developers are familiar with all possible deployment platforms. However, it would be great to have a set of base conventions that platform-specific tools can extend in different ways. For example, there could be a base convention that documentation lives in a docs/ subdirectory in Markdown format, and the build-posix-package tool could have an extra convention that a file named docs/manpage.md will be installed as the manpage for the crate's main executable.

russel commented 6 years ago

I got pointed at this issue by @killercup on https://users.rust-lang.org/t/cargo-and-installation-of-ancilliary-files/15974

My specific problem is installing a GNOME application. So not a CLI application per se, but very similar. Whilst many of the resources needed by the application are embedded in the executable, there are some that need installing. Manual pages and user documentation certainly, but also some image files and a GNOMEShell launcher.

Whilst my need is very specific, I am sure it acts as a microcosm of the more general situation.

Meson makes it very easy to install application – my experience is with D and C++, I have not tried using Meson with Rust. It is also fairly straightforward, but not always easy, to set up installation using SCons and CMake – but again my experience is with D and C++, to date I have only used Cargo with Rust.

kbknapp commented 6 years ago

@Screwtapello: I suspect no single tool is going to be able to support many or varied platforms, because platform conventions vary so widely, and because so few tool-developers are familiar with all possible deployment platforms.

This is my exact concern. I agree very much with the rest of your comment that having separate tools or conventions that people can pick/choose/extend would go a lot further than an "all in one" tool. This is with the caveat that those "separate tools" are well known, widely supported, well documented, etc. Not just one offs.

@jaemk: For distributing my cli apps I made https://github.com/jaemk/self_update to support a "self update" command that pulls the latest github release built using https://github.com/japaric/trust/

I love this! It may not work for everyone, or for complex use cases, but making it easy to have a "self update" feature of Rust CLI applications for "small utilties and binaries" would probably go a long way. Of course this doesn't solve the initial installation piece, but I still think it'd be great to include in a full blown solution. This particular piece could just be a standard guide on how to do this, such as a cookbook like recipe.

[about cargo-install in general as an install tool]

I'm on record saying I don't really like leaning on cargo-install for the general public, or general purposes. I think it's great for dev tools, or things that make sense to use during Rust development, but I personally don't want to expand it's use case to a general purpose installer.

I think there's too many nuances, and too much complexity to do everything correctly for it to all be contained in cargo. I'd like to keep cargo a Rust build tool, not a general purpose package manager. This isn't to say we shouldn't add certain features to cargo-install such as post build scripts...I'd just like to have proper guides/tools doing things the correct way first so that people don't just hit the easy button of cargo install which is what people are starting to do already.

Why?

Here's a few of the things I think would need to be sorted out (conversely why I think they shouldn't sorted out). This isn't meant to be an all encompassing list, just a quick search of some relevant issues. There may be more I can't think of off the top of my head, but here's a few:

Advanced or Post Build Steps

My primary concern with using cargo install too generally is that because of all the added complexity of correctly installing software, cargo-install takes the easy route and doesn't support complex/advanced use cases. Which is perfectly fine to me because it's not a general purpose installer. Until recently cargo-install didn't even support use of Cargo.lock files. As far as I know it still doesn't (easily) support more advanced build steps (like setting RUSTFLAGS, or like many have mentioned post build steps).

Crate Size

Including things like docs, manpages, completion scripts, licenses, axillary files, etc. which will be required for installation of end user products will bloat the crate packages. I'd actually like to do the opposite of de-bloating crates

Binary Only Features/Deps

Until cargo supports binary only features and deps, crates that include both a lib and bin will suffer from the above points when used as a lib (compile times, crate size, etc.)

Also, being able to specify binary dependencies is huge for a general purpose package manager.

Code Signing

This is another pretty big hole if we'd want to properly use cargo install as a general purpose installer.

vitiral commented 6 years ago

Just to include packagers, nixpkgs has pretty good support for rust ATM. I managed to get the novault nixpkg created, which required custom external dependencies.

The advantage of nixpkgs is that they can run alongside other package managers on any Unix distribution (including Mac) and I believe they are trying to support windows as well. This allows for at least some installation path for almost any platform/distribution, which is very valuable.

The process is pretty straightforward, with carnix creating the package from the Cargo.lock file. I think it's a workflow worth emulating for other packaging scripts as well.

matthiasbeyer commented 6 years ago

Just to include packagers, nixpkgs has pretty good support for rust ATM.

Tell me more about this! The last time I tried to package something for nixpkgs, I failed hard. I used carnix and it created a package expression which OOM-Errored with Nix!

From what I think, Rust packaging with nixpkgs is hell!

vitiral commented 6 years ago

@matthiasbeyer this might fix your problem (just came across it): https://github.com/NixOS/nix/issues/358

Edit: link to PR

luser commented 6 years ago

cargo install is nice for developers who already have Rust + cargo installed and are likely to have whatever -dev packages are required to build the app, but it's a bit heavyweight otherwise. Especially since you can wind up pulling in a lot of dependent crates to build a useful tool, which means the build time for your tool creeps up...

It would be nice to have best practices for alternatives as well. For sccache we've got binary releases being built in Travis + Appveyor for Mac/Windows/Linux: https://github.com/mozilla/sccache/releases

Getting this right involved a bunch of fiddling which I'd be happy to help document somewhere: https://github.com/mozilla/sccache/blob/master/.travis.yml https://github.com/mozilla/sccache/blob/master/appveyor.yml

epage commented 6 years ago

@luser I'm planning on documenting creating binary releases on my crate-ci docs

See https://crate-ci.github.io/

I put it on hold for the moment because I felt like documenting the process was a bit much. Because we need to know what the target is to find the binary, we have to retrofit the CI configuration in a fairly invasive way.

My hope is soon(ish) to start a tool to make it easier to get the files needed to make it easier to document See https://github.com/crate-ci/meta/issues/1

XAMPPRocky commented 6 years ago

@kbknapp I think what you've listed as reasons cargo shouldn't be used as a build, is exactly the reasons this working group should push for solving those problems as anything that hampers general development also hampers the development of developer tools. Saying "I'd just like to have proper guides/tools doing things the correct way first so that people don't just hit the easy button of cargo install which is what people are starting to do already." is honestly a bit elitist, if people don't have an easy way they won't work on the tools. No amount of documentation or templates can compete with a single line command.

The point of this working group is provide the easy way for developers to release and distribute their apps saying "Oh well you need to integrate in with this CI template, and add this dependency to be able for users to upgrade your app." isn't an easy way to release apps, it's tedious busy work. So what if the program is only available to Rust developers? That's a great way to get feedback, you can release a 0.1 through cargo and go to r/rust or IRC and ask them to try it out.

Unless you can somehow automate the process of releasing to all package managers at once with zero configuration any other option is worse than releasing through cargo-install. Saying that because cargo lacks features so it shouldn't be a package manager is a self fulfilling prophecy as we're the CLI working group and if we choose not to try to improve cargo as a package manager that will be used as reasons to not improve cargo-install.

spacekookie commented 6 years ago

@Aaronepower Well, that's all well and good and yea maybe there are some things in the cargo install pipeline that aren't very nice right now. But to target that at end-users is the wrong way to go about it.

Writing apps and CLIs specifically in Rust is so great because you're not dependent on any interpreter or runtime or toolchain. And then to say: install this toolchain to install the application is a bit…weird.

If there are any hangups with cargo-install then yea, we should of course fix them. For example a super-duper awesome feature would be that sys- package automatically install the required -dev packages on your platform so that compilation doesn't fail if they're not there.

But as an end-user experience, I don't want them touching cargo at all

XAMPPRocky commented 6 years ago

@spacekookie I feel like that is bit too much of generalisation on end users and the applications being made. Obviously if you're making a tool that helps write python applications it'd be a pain point to require cargo, and if you're a project that as big as ripgrep you need to provide other avenues of installation.

A lot of projects are Rust focused and even if they aren't when they start out they need a way to easily distribute their application to get feedback. I'm saying that even just as a quick way to get dev tools or small cli apps cargo-install needs to be held to a higher standard as I think improving that will have the most impact on developers releasing their applications.

artem-zinnatullin commented 6 years ago

Should we split this issue into two separate issues then?

One issue could focus on making Cargo more friendly for installing CLI tools written in Rust with Rust-dev community as target userbase.

Another issue could focus on documenting distribution of CLI tools written in Rust with different package managers (+ OSes and architectures) like Homebrew, apt, etc with regular non-Rust dev audience as target userbase.

killercup commented 6 years ago

This issue was never meant to cover cargo-install 😅

So, yeah, please go ahead and make that issue! Ideally with quotes from the comments here :)

artem-zinnatullin commented 6 years ago

@killercup if it wasn't, maybe you could update issue description to specify that it tracks distribution through package managers (or other solutions) to final non-Rust-dev users?

That'll help avoid confusion we had here :)

ayosec commented 6 years ago

For Linux systems, it may be good to choice either Snap or Flatpak, instead of trying to support all available package managers.

spacekookie commented 6 years ago

@ayosec We don't have to support all package managers.

Creating source deb files, source rpm files would be a great start. Yea sure, making snap and flatpak packages is also nice. But not everybody likes those approaches (me included :stuck_out_tongue_winking_eye:), so supporting "classic" Linux packages is also good.

Besides, deb (for bin packages) already exists :+1:

joshtriplett commented 6 years ago

I do think that it's worth separating out two separate considerations here, as a few people have suggested.

We want to support building and installing CLI tools and all the accompanying bits that people expect, including manpages, completions, etc.

We also want to support building policy-compliant source packages for Linux distributions, so that distributions can ship Rust software just as easily as they can ship anything else.

tblair commented 6 years ago

I know I'm a bit late to this discussion, but I looked into what it would take to build a Cargo extension to package up a CLI app a while back and thought I'd add a bit of what I came up with after researching the issue.

It's great that this issue is getting attention and I wish I had more time to pitch in and help.

artem-zinnatullin commented 6 years ago

@tblair

Creating a very basic installer is pretty easy on most platforms, but for more complex options that make installers feel professional like adding a logo or installing files beyond the single executable, the developer will need to supply additional metadata that's currently awkward to put in Cargo.toml.

Hm, do we need to have logos in CLI tools installers? Do we need installers for CLI tools?

Homebrew should not be looked at as an acceptable Mac install solution, or at least not the only solution. It's much more analogous to cargo install in that it's targeted at developers and other more-technical users. The likely best/default answer on Mac is a .pkg file assembled with the pkgbuild tool.

Well, CLI tools are mainly used by developers, sysadmins, devops and pro-users who tend to use some package manager and Homebrew is one of the most popular ones for macOS 😸

However definitely agree that it shouldn't be the only option 👍

Windows .msi and Mac .pkg are particularly important because they fit in with the first-party remote administration tools that IT departments use to provision their fleets of machines and keep them up to date.

Interesting perspective! // Curious though how often do CLI tools need to be provisioned on Windows and Mac by IT departments, hmmm. But still definitely agree that OS-native packaging like .msi and .pkg must be covered anyway.

epage commented 6 years ago

I've updated the issue with a summary of the conversation in context of the what we are working to achieve with this working group

I've left off man pages because generating them is more tangential to this discussion and I'll fork it off into a separate topic

@spacekookie

@killercup When it comes to automatically generating debian packages (and there is already cargo-deb for that) I think it's even more important to generate source packages for distributions to include (or to build a PPA with). And that is something that cargo-deb can't yet to.

@joshtriplett

We also want to support building policy-compliant source packages for Linux distributions, so that distributions can ship Rust software just as easily as they can ship anything else.

Is packaging for a distribution usually handled upstream or within distribution community? I worry that there are a lot of constraints and requirements in trying to cover this use case that it'll overwhelm us for what we want to accomplish this year.

@ayosec

For Linux systems, it may be good to choice either Snap or Flatpak, instead of trying to support all available package managers.

@tblair

Windows .msi and Mac .pkg are particularly importan

I've not used Mac pkgs, Snap, or Flatpak. Do they easily support running from the command line, like putting their bin in the path?

joshtriplett commented 6 years ago

@epage

Is packaging for a distribution usually handled upstream or within distribution community? I worry that there are a lot of constraints and requirements in trying to cover this use case that it'll overwhelm us for what we want to accomplish this year.

It's something with many common requirements across distributions, so some aspects of it need addressing upstream, in collaboration with the distribution communities. We need to take those requirements into account in our development.

pickfire commented 6 years ago

@ayosec @spacekookie If we only support certain distributions, what about the rare cases such as Arch and Alpine?

What I think is that we can first identify the common dependencies for different distributions and a distro-based package generator and probably automate the distribution of rust packages on releases.

With that we can support new distribution that may be popular in the future instead of having limited to apt, rpm and the big guys. Not sure if this is feasible though.

epage commented 6 years ago

@soc pointed out https://sbt-native-packager.readthedocs.io/en/stable/

It has some similar concepts to what I am looking at creating. Initially, I planned for the user-configuration to be siloed by the packaging targets but combining information, where possible, is possible in the my scheme and might be considered in the future.

If we instead adopted sbt, we'd still need to develop a way to make it easy to find the binary (see https://github.com/crate-ci/meta/issues/1)

soc commented 6 years ago

@epage I'm not recommending adopting sbt, but rather having a look at the concepts and figure out which parts work for Rust. I think people are largely happy with how sbt-native-packager works, so it could make sense to check any design decision against it and evaluate whether its an improvement, a different approach, or something that could be improved upon.

epage commented 6 years ago

@soc ok. My companies installers are actually similar and what I was modeling things after already. I do need to actually look to see whats different rather than just assume they are the same.

berkus commented 6 years ago

flatpak all the things maybe? :)

soc commented 6 years ago

@berkus I'd rather prefer avoiding Flatpak, or at least supporting the "standard" means of distributing packages (deb, rpm, ...) as a preferred option.

It really irritates me is that the Flatpak developers not only knowingly violate the XDG base directory spec, but also try to obscure the fact that they are doing it.

It took half a decade to get to a state where one can reasonably assume that applications follow the standards of the operating system they run on. I have zero appetite for new applications regressing on that.

YMMV, though.

gibix commented 6 years ago

There is also also a gentoo support for generating ebuild recipe from a cargo project: https://github.com/cardoe/cargo-ebuild.

spacekookie commented 6 years ago

(slightly off topic) @soc not to mention that things like GUI applications packaged with flatpak will just ignore your gtk theme and use whatever the developers bothered to include (usually the windows 95 fallback)

soc commented 6 years ago

(offtopic) @spacekookie +1. The current state is just sad, and Flatpak doesn't do much to improve it, despite hijacking XDG environment variables – which is just a terrible thing to do. Things would be so much nicer already if some distribution just declared "we are making $HOME read-only, fix your applications and stop dumping random files into the folder", and then people could use this as a starting point to further sandbox applications, making them easier to package and distribute.

epage commented 6 years ago

For those interested, I've created a doodle for getting a real time conversation going about how we can align and share within the different packaging tools https://doodle.com/poll/6x2wgpxznaf7b7g5

epage commented 6 years ago

RE homebrew

@uwearzt pointed out https://github.com/Sean1708/cargo-brew but it just wraps cargo install which is limited.

@rcoh has a template for brews https://github.com/rcoh/rust-brew-template

We talked about turning that into a crate-ci example and adding documentation for it in crate-ci docs.

Ideally, we'd also create a cargo brew (name pending since one exists) that will generate and update the tap files as needed.

kornelski commented 6 years ago

I've not used Mac pkgs […]. Do they easily support running from the command line, like putting their bin in the path?

On macOS there's command-line installer that can install .pkg headlessly without user intervention. It has parseable output. Mac Sparkle updater uses it to apply complex updates.

eberkund commented 5 years ago

Are flatpak / snap's apps added to PATH?

I know snaps are.

jayvdb commented 4 years ago

In Open Build System, there are currently 282 crates already packaged. There wiki page about Rust only documents the bootstrap, not how to build crates, but looking at the spec it seems they are using rust2rpm.

Fedora has ~820 rust-* specs, so they are even further along, and that should be nearly enough that each CLI app only needs to add packaging for one or two missing deps before the CLI can be built.

matu3ba commented 4 years ago

(offtopic) @spacekookie +1. The current state is just sad, and Flatpak doesn't do much to improve it, despite hijacking XDG environment variables – which is just a terrible thing to do. Things would be so much nicer already if some distribution just declared "we are making $HOME read-only, fix your applications and stop dumping random files into the folder", and then people could use this as a starting point to further sandbox applications, making them easier to package and distribute.

That may only work, if you can like google with Android force them to simplify/let the user decide how to organize their (user)configurations, (user)data, (user)install setups, caching and the central lookup of these information. This would mean 1.having a standard how to write these data (as file format) + 2.sandbox enforcing it + 3.tool for import/export/moving of datafolders.

Android/Apple etc are quite inflexible regarding structure, so something similar should not work.

I will suggest something like this to distri.