Make Chapel installation easier

[ronawho]

Can we build and release binary packages ourselves and provide a one-liner to do installation?

Today, our standard install requires downloading the tarball and building from source. We have binary distributions for Crays (as a Cray module) and for Mac (through homebrew) but we don't have straightforward binary installations for *nix platforms.

We have put some effort into trying to get a Debian package:

• https://github.com/chapel-lang/chapel-packaging/tree/master/debian
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880991

But the main disadvantages there are that Debian has pretty strict packing rules (and we're very far from packaging experts.) This means we can't bundle any of our third-party, so in addition to it being a lot of work on our end, we'd still only end up with a minimal quickstart installation of Chapel, which would only be useful for playing/experimenting but performance would be bad. Additionally, since our performance and stability are improving pretty dramatically with each release we want people using the latest release, which is pretty hard to do with the lead time for updating official Debian packages and having those trickle downstream to Ubuntu and other distros.

So for this issue -- can build and distribute our own binaries for platforms and architectures that are most important to us and how can we make it simple for users to install and update?

[ronawho]

Based on cursory look -- I like what rust does. Their typical install is a one- liner:

curl https://sh.rustup.rs -sSf | sh

This downloads and runs a small sh script which detects your platform and downloads rustup and runs the rustup installer. Rustup is a toolchain multiplexer, which basically allows you to have multiple different versions installed and easily switch between them. They have stable, beta, and nightly release channels:

New nightly releases are created once a day. Every six weeks, the latest
nightly release is promoted to ‘Beta’. At that point, it will only receive
patches to fix serious errors. Six weeks later, the beta is promoted to
‘Stable’, and becomes the next release of 1.x.

This process happens in parallel. So every six weeks, on the same day,
nightly goes to beta, beta goes to stable.

This seems like a really good setup. It's easy to get the latest stable release and keep it up to date, which is great for people wanting to use the official release. Or for people who typically work off of our git master today, they could work off of nightly for the latest bug fixes. And for all of this there are binary releases. For me it took less 3 minutes from starting to download the rustup-init script to having a working installation. Here are some other rustup resources that I found useful:

• https://www.rust-lang.org/en-US/install.html
• https://www.rust-lang.org/en-US/other-installers.html
• https://github.com/rust-lang-nursery/rustup.rs

[ronawho]

And note that we have some unique challenges for doing binary releases, most notably with GASNet, which really wants to be configured/built on the system it will be used on. See the thread in https://github.com/chapel-lang/chapel/pull/10175 and possibly https://bitbucket.org/berkeleylab/gasnet/pull-requests/36/wip-build-shared-libraries for more info.

Still as a first step even single-locale binary release would be pretty great

[ben-albrecht]

I wonder how difficult it would be to create a Spack package for Chapel? It looks like they already have a qthreads, gasnet, and hwloc package (though I'm not sure we'd use those due to challenges you pointed out).

Additionally, I know @mppf had some thoughts about the proposed one-liner solution as well as other ideas about deploying Chapel via snap packages.

[mppf]

I wonder how difficult it would be to create a Spack package for Chapel?

I think creating a Spack package for Chapel sounds like a great option. I think it's highly likely we can get Spack to handle the various included third-party packages in a sane way (whether that's through other Spack packages or not).

@mppf had some thoughts about the proposed one-liner solution

Maybe I'm old-fashioned but I can never get comfortable running curl | sh type scripts. Just seems to set a horrible security precedent, even if it's OK in the specific case. In other words, it's training users to do all the wrong things to achieve security.

That doesn't mean we can't support such a 1-liner; it just means that I'll find it deeply unsatisfying & I hope it's not the entire plan. Nonetheless I'd prefer it if we just never created such a thing.

I don't see anything wrong with creating .deb / .rpm / whatever binary packages. These are generally code-signed. We've already done the work to create a Debian package and I don't think it'd be particularly hard to move from that to nightly .deb builds. I think a snap package would be an interesting alternative in this general area.

Besides all of that, for the record, I think we should continue to try to create a Debian package. I have no idea when we'll be in a Debian distribution, but if that's a goal, and it takes time, we should keep trying (at a low level of effort).

[ronawho]

I'd love to see a spack package and/or binary packages. This is probably just ignorance on my part, but what's so insecure about the curl | sh install? What makes that any less secure than say grabbing pre-built binary-packages from us? FWIW rust and nim provide curl | sh installation options (and it's the default install method.) In that case I think the curl command grabs rustup or choosenim respectively, which will then go grab the appropriate binary packages.

I really just want a fast installation of Chapel that isn't limited to quickstart (i.e has 3rd parties including llvm built) with minimal steps for the end user. I don't have a strong preference for the curl one-liner or something better (and I don't really know this space that well anyways.)

FWIW I don't see the benefit in pushing on an official debian package in the near or medium term. Even if we get something accepted it's going to be quickstart only and it will be a year or so out of date by the time it's in downstream distros. With how much we're still changing I think we want something more flexible that makes it simple for users to use the current release.

[mppf]

In the "better late than never" category, here is my answer to this question:

what's so insecure about the curl | sh install? What makes that any less secure than say grabbing pre-built binary-packages from us?

If you do curl | sh then you can't inspect the commands that you are going to run (potentially as root, depending on the script setup). You can't check its hash and verify that you are getting the published version. You can't check it for code signing.

You might be tempted to run the curl command to inspect the commands and then run curl | sh to run it. But you have no guarantee that the server will return the same commands twice.

In contrast, if we have people download a file and then do something to install it, inspecting/verifying checksums/verifying signatures are possible. And, even better, it's my understanding that .deb and .rpm packages include code signing that is checked automatically.

(Note that here I'm talking about SHA-256 checksums, say).

But the curl | sh pattern is worse than that. It creates a single point of failure in terms of security for an entire community. What if the webserver hosting the files that we are downloading with curl is hacked? It will still be itself, so HTTPS will offer no protection. But it could be hacked to send a completely different script that does nasty things. And if normal practice is not to inspect the script, would anybody even notice?

In contrast, things that include code signing or at the very least checksum verification make such problems less likely, because the checksum and/or the public key can be published in multiple places & as a result are harder to impersonate. For example, a web server showing the link to download can also show the checksum and link to a different server for the actual .tar.gz file.

I personally avoid curl | sh because I want to be able to look over the script I am running. I'd rather check a checksum, but people don't usually publish checksums for such scripts, so looking over it is all I can do. But I do verify checksums when I download a source release (e.g. a .tar.gz file).

[mppf]

To be clear, I think it's possible to have a paste-able command (that might be technically "one line") that addresses my concerns above.

[jabraham17]

Following up on this issue, we now have support for both singlelocale and multilocale packages. This provides a 2-liner way to install chapel, for example on x86 ubuntu24

wget https://github.com/chapel-lang/chapel/releases/download/2.1.0/chapel-2.1.0-1.ubuntu24.amd64.deb
sudo apt install ./chapel-2.1.0-1.ubuntu24.amd64.deb

I am going to leave this issue open to continue to capture the desire for a rustup-like command