Add windows gui / msi installer [difficult]

[brson]

To complete the Rust installation experience on Windows we want to be installing rustup via an msi.

Make a proof-of-concept rustup msi installer that embeds libmultirust. The behavior of this msi will be heavily customized - all it does is the standard rustup install, but presented in a windowsy way. To start with it can be really simple:

• On installation, present a screen that says more-or-less what the console installer says now. Have install/cancel buttons.
• After they press install, go to another window that displays whatever status updates we reasonably can as the install is progressing.
• After install show another screen that indicates success.
• Register the uninstaller with windows so it works from the add/remove programs screen.

This will require a lot of refactoring of the existing install code to get it embedded in this new context.

Our options for GUI's will be limited but we can't use something heavy. I'm thinking either something rust-centric like conrod, or just a very thin win32 wrapper.

[retep998]

Depending on OpenGL for Conrod might not work very well if they don't have the GPU manufacturer drivers that provide OpenGL, or if they're using Windows in a VM.

[Diggsey]

We could just use Inno Setup... It will allow us to produce a GUI installer which can call out to the command line rustup behind the scenes, and it's configured declaratively, so we won't even need any non-rust code.

[brson]

@Diggsey Inno Setup does not produce msi's (I believe), and msi's are the delivery format people most expect and prefer.

Rust itself uses Inno to produce .exe installers but we don't advertise them because the .msi's are nicer.

[retep998]

If we use the Windows Installer stuff then that can provide a user interface for us.

[pravic]

@brson

and msi's are the delivery format people most expect and prefer.

.msi is a really big pain due to windows installer cache bloat :(

[vadimcn]

If it's just a window with some text and [Install]/[Cancel] buttons, the built-in msi UI will do just fine. Anything more complicated/dynamic than, say, the Rust installer, would be quite annoying to do in it, and in that case I would suggest going with WinForms/WPF.

I also want to note that invoking an .exe to perform the actual system changes (like mucking with the registry to modify PATH) goes against the "best MSI practices", but I guess we'd be fine with it for cross-platform consistency's sake?

[Diggsey]

@vadimcn Modifying the PATH would have to be done in a custom action anyway, even if .msi is used, so there's no real benefit in that regard. Aside from that, rustup doesn't actually make any changes to your system (even rustup itself goes in your user folder)

[vadimcn]

Modifying the PATH would have to be done in a custom action anyway, even if .msi is used,

Why? MSI has direct support for modifying environment vars. But even when CAs are used, the recommended way is not to perform system changes directly, but rather to schedule them for execution by the MSI engine in elevated part of the install process (by adding ephemeral records to File/Registry/etc tables).

[brson]

@vadimcn I don't want to invoke an exe.

What I want to do is have the msi system call functions in the multirust dll to perform the installation actions, while presenting the UI people expect from an msi installer. Can we do that?

[vadimcn]

What I want to do is have the msi system call functions in the multirust dll to perform the installation actions, while presenting the UI people expect from an msi installer. Can we do that?

Yes, that is possible. (This still counts as performing changes directly, though)

[nodakai]

In any cases, please keep supporting non-Administrator users installing Rust to somewhere under C:\Users\username or even C:\Users\Public

[Boddlnagg]

You may also try using WiX (http://wixtoolset.org/), it's a declarative way of creating a Windows Installer (msi) that can be heavily customized. It also says that it supports custom actions written in C++, so it shouldn't be too hard to use custom actions written in Rust. On the other hand, however, this might be too big of a non-Rust build system/dependency.

[jminer]

As a user, I've always preferred Inno Setup or NSIS installers over msi. Most msi installers I've used are slower and don't have as nice of a UI as the open source installers. They've also been buggier, but that might not be the Windows Installer's fault. I really don't want a UI made with conrad or .NET. Conrad does not look native and either one would add overhead.

I've written an NSIS installer before with similar screens, and I know it can call C functions. I'd be willing to work on an NSIS installer if there was a chance it would be used.

[nxnfufunezn]

How about this https://github.com/andlabs/libui

[tiborgats]

How about using Qt Quick ? It is more mature than libui, has better documentation, the problem of high dpi (4K UHD) screens is solved too.

[nxnfufunezn]

@tiborgats that would add an unnecessary dependency for QtQuickControls etc for a simple installer.

[brson]

I'd like to keep using WiX for this. Seems to be the most 'modern' choice. It's what we're using today.

[brson]

The way to get started here is just prototyping: figure out how to make WiX, the rustup library and the Win32 GUI APIs work together to present something that looks plausibly like an installer.

[Boddlnagg]

I tried to get this working and have a prototype running that ...

• uses WixUIExtension (built-in UI that looks like standard windows installers, and can be customized e.g. with a page for rustup install settings), so there's no need to use Win32 GUI APIs.
• runs a custom action from a Rust library (cdylib), which currently only reads a property from WiX (so we can read settings that are changeable in the UI) and logs it to the MSI log.

The only issues I encountered were the following:

• When building the DLL with the custom action I had to link some libs from the WiX SDK and build with a 32-bit compiler.
• Windows Installer requires something to be installed by the installer itself (either a file or a registry key), so I chose a registry key under HKCU\Software\rustup. Maybe the %USERPROFILE%\.rustup directory can be used instead (but that probably means that the installer will fail when it exists).

I have uploaded my experiments here: https://gist.github.com/Boddlnagg/9d8f01e6d844cd78473651470282ebda

I might be able to work more on this over the next weeks, but I don't want to keep anyone else from doing so, when they are faster.

[Boddlnagg]

Further investigations led me to the MsiProcessMessage function (though it is better to use the WcaProcessMessage version from wcautil), which can be used to send progress and status updates from the custom action back to the UI. This requires the custom action to run as deferred (instead of immediate), which is the right thing to do anyway. I have updated my gist to use deferred.

The rustup installation routines will have to be refactored in such a way that status updates can be either reported to the console or via WcaProcessMessage.

[brson]

@Boddlnagg thanks for doing that research! Using a registry key for registration seems just fine.

When building the DLL with the custom action I had to link some libs from the WiX SDK and build with a 32-bit compiler.

This seems ok. We can use a 32-bit installer everywhere.

The rustup installation routines will have to be refactored in such a way that status updates can be either reported to the console or via WcaProcessMessage.

Makes sense.

In order to get the self-install to work from a library we're going to have to do some refactoring of the project structure. Right now there are two top-level artifacts: the rustup library and the rustup-cli binary. The rustup-cli bin is both rustup-init and rustup - it changes behavior based on how its invoked. The windows installer though is going to need access to that binary via a library. So we're going to have to create a new library that include!s the entire rustup binary. I'm not entirely sure how to make this happen but there are going to be at least a few steps.

• To start with I'd say we just create a new project rustup-win-installer in src/. This project will depend on the main project. To start with it can just expect that the rustup binary exists in a known location, since getting a dependency to output a binary is not possible.
• The self_update module in rustup-cli will require significant refactoring:
  • The code needs to move into the library rustup
  • The console specific bits need to be lifted out and injected from rustup-cli
  • The routine for finding the rustup binary likewise needs to be lifted out and provided by injection, since the win installer will be getting it from an included binary, not from the running executable.

With those refactorings we can whip up a basic installer that presents no options but does put the stuff in the right place.

[brson]

(I can probably help with the self_update refactoring since it could get ugly).

[Diggsey]

To avoid duplication it's technically possible to load an executable as though it were a DLL and call functions from it - http://www.codeproject.com/Articles/1045674/Load-EXE-as-DLL-Mission-Possible there's probably a cleaner way though.

[Boddlnagg]

I skimmed over https://github.com/rust-lang-nursery/rustup.rs/blob/master/src/rustup-cli/self_update.rs and was thinking that maybe we could remove the complex Windows-only logic to remove the running exe, by always using the MSI for uninstall/update (installed MSIs are cached, so it will exist on the system). That requires that rustup has been installed using the MSI, but if that will be the only method on Windows (i.e., rustup-init will no longer exist), it should be fine. I don't know about a possible upgrade path for already existing installations that have never used the MSI, though.

Also some other routines, that have Windows-specific codepaths (such as updating the PATH) could be handled by MSI directly. WiX provides ways of doing this easily, since it's something that installers often do.

[retep998]

@Boddlnagg Just tell the user to delete their rustup installation and download and install using the MSI instead?

[Boddlnagg]

@retep998 That would be possible, of course, but will it be convenient enough?

@brson I don't quite understand these sentences:

To start with it can just expect that the rustup binary exists in a known location, since getting a dependency to output a binary is not possible. [...] The routine for finding the rustup binary likewise needs to be lifted out and provided by injection, since the win installer will be getting it from an included binary, not from the running executable.

Why are you talking about the rustup binary? I was expecting that the MSI just includes the custom action DLL (cdylib), which links everything in statically, so no binary would be included. Update: Oh well, I think I got it now ... the rustup binary is the thing that will be installed, after all, so it must be included. But then it should be extracted by the MSI itself. Is the place where it should be put known in advance or does some Rust code need to run to determine that?

[brson]

I skimmed over https://github.com/rust-lang-nursery/rustup.rs/blob/master/src/rustup-cli/self_update.rs and was thinking that maybe we could remove the complex Windows-only logic to remove the running exe, by always using the MSI for uninstall/update (installed MSIs are cached, so it will exist on the system).

I think this is a reasonable goal to aim for, though perhaps we can get there incrementally. If it ends up just being a lot easier to implement then maybe we can jump straight to that model. The transition story can be worked out later - there's a lot of problems to solve just to get to the point where we have a working GUI installer.

Update: Oh well, I think I got it now ... the rustup binary is the thing that will be installed, after all, so it must be included. But then it should be extracted by the MSI itself. Is the place where it should be put known in advance or does some Rust code need to run to determine that?

It's not obvious to me that it should be extracted by the msi itself since there is other logic to installation than just extracting the binary, but if we can make it work that way then that's probably best. The logic for deciding the installation is in self_update::install_bins and utils::cargo_home but it's pretty simple - if CARGO_HOME is set put it there; if not put it in a pre-determined location. This logic will likely expand somewhat in the future.

The bulk of the GUI customization work we'll need to do is for configuring the global installation options and for installing/updating/uninstalling toolchains. If the basic work of putting the rustup bin in the right place can be done by the MSI system itself that seems good.

[Diggsey]

How are updates going to be handled with an MSI installer? If we just use the existing update system, then 1) it might confuse the uninstaller if the files are different from those it installed, and 2) windows will show incorrect version info in the list of installed software.

However, requiring the user to download a new .msi each time isn't great, but I imagine you could have rustup download and run the .msi? Or there might be some facilities for updates built into the windows installer system, I don't remember.

[retep998]

I use some software which uses a .msi to self update. All it does is trigger the UAC prompt and then I see a small progress dialog and then its done. So it is definitely possible to have a self update using that.

[cyplo]

The standard way of doing self-update is called ClickOnce not sure how hard it is to do now with just free tools, needs some research.

[retep998]

ClickOnce

[Boddlnagg]

ClickOnce seems to be not an option in this case. (1) It's an alternative to Windows Installer, so you can't use it update an application that has been installed using MSI and (2) while it might be fine to use only ClickOnce, it seems that "ClickOnce installs files to an obfuscated folder in the current user's profile. You can't have it update files all over the file system." (source)

For self update, I would suggest that running rustup self update just downloads the most recent msi (if there is an update) and runs that. It could be done in silent mode, so the user won't even see a GUI in that case. Apparently you can also pass a URL to msiexec directly: https://msdn.microsoft.com/de-de/library/windows/desktop/aa368328(v=vs.85).aspx

[cyplo]

Hi ! re (1): Haven't tried it myself but it seems possible to update app installed via MSI using ClickOnce; by the way of making MSI-installed app a 'prerequisite' to installing the ClickOnce app. Probably from now on you would have to use ClickOnce for updates solely though (?).

Not pushing for ClickOnce by any means here, I just know from experience that this is the most familiar interface for people distributing and receiving self-updating apps on Windows.

[Boddlnagg]

It's not obvious to me that it should be extracted by the msi itself since there is other logic to installation than just extracting the binary, but if we can make it work that way then that's probably best. The logic for deciding the installation is in self_update::install_bins and utils::cargo_home but it's pretty simple - if CARGO_HOME is set put it there; if not put it in a pre-determined location. This logic will likely expand somewhat in the future.

In that case, I would suggest the following procedure in the MSI:

• Run a custom action before InstallFiles that determines the place to put the rustup binary
• Let the built-in InstallFiles action handle the extraction of rustup.exe
• Run a custom action after InstallFiles that does all the rest (create hardlinks (not possible from WiX), install default toolchain, ...), except adjusting the PATH
• Adjust the PATH from WiX

I have implemented the basic structure already in my updated gist.

The next step would be to create an actual PR with the rustup-win-installer project in src/ as proposed by @brson. However, there will be two projects:

• The embedded rustup library with the entry points for the custom actions (a Rust project with cdylib target).
• The actual installer, which is built using the WiX tools (no cargo/rustc involved). But I don't know how to invoke the WiX tools in such a way that they work in the CI setup (the Rust packaging uses a Makefile). The installer will depend on the rustup library above and the rustup binary.

[Boddlnagg]

@retep998

I use some software which uses a .msi to self update. All it does is trigger the UAC prompt and then I see a small progress dialog and then its done. So it is definitely possible to have a self update using that.

I managed to replicate this behavior (modulo the UAC prompt, which we don't need):

msiexec /i https://<host>/rustup.msi /passive

The /passive switch disables user interaction, but still shows a small window with a progress bar for downloading, then another one for installing. So rustup self update can just execute that command and exit immediately so that it won't run when the installer tries to update rustup.exe.

For uninstalling, the code in my gist now writes the (automatically generated) ProductCode to the registry, so rustup self uninstall can fetch the product code from the registry and run msiexec /x "<product-code>". This will run the uninstaller for the cached MSI. I also checked that the cached MSI (in %WINDIR%\Installer) will be deleted after uninstall, and the registry entry will be gone, so this should leave no traces behind. Of course the user can also uninstall directly from the Windows Control Panel list of installed software.

[skade]

Is installer signing also a necessary feature?

https://github.com/rust-lang/rust/issues/25457

[retep998]

@skade yes

[brson]

However, requiring the user to download a new .msi each time isn't great, but I imagine you could have rustup download and run the .msi? Or there might be some facilities for updates built into the windows installer system, I don't remember.

I'd expect rustup self update to download the msi and run it non-interactively; and for rustup self uninstall to do the same, probably with custom logic to delete toolchains etc.

The actual installer, which is built using the WiX tools (no cargo/rustc involved). But I don't know how to invoke the WiX tools in such a way that they work in the CI setup (the Rust packaging uses a Makefile). The installer will depend on the rustup library above and the rustup binary.

To start with I suggest a bash script, or maybe powershell or python if you prefer. The CI can run it explicitly after cargo build.

@Boddlnagg This all sounds awesome.

[brson]

@Boddlnagg if you find yourself needing to make changes to rustup self update to support the new uninstall then can I suggest doing it under a feature flag for now so we can continue producing rustup's with the current model while we iterate on the new installer?

[Boddlnagg]

@brson Sure, I will use a feature flag. But I won't be able to do much now until the end of next week, just as a heads-up. (Maybe someone else wants to chime in and already start with the refactorings? :wink: )

[brson]

I'll put it on my todo list to start refactoring the installer code, but suspect I won't get to it this week.

[Boddlnagg]

I just went through self_update.rs and tried to identify those parts that need to be shared by the MSI installer. To be able to reuse them, those parts will need to be moved from the executable into the library (and potentially refactored and adapted).

• [ ] do_pre_install_sanity_checks() (maybe offer to uninstall MSI-installed Rust automatically)
• [ ] cleanup_legacy()
• [ ] Creation of hardlinks, currently part of install_bins()
• [ ] maybe_install_rust()
• [ ] uninstall(), but it should not modify PATH, should not remove rustup.exe, but should remove the hardlinks (MSI will modify PATH, and delete rustup.exe and the containing folder)

[brson]

Thanks for doing the analysis!

[Boddlnagg]

I opened an initial WIP PR at #635.

[brson]

Thanks @Boddlnagg. Great progress.

[brson]

We've merged the first iteration, which puts the basic structure into place. Thanks @Boddlnagg!

It's even set up to build on appveyor, but there's a bit left to do. What I know of:

• The msi-ified rustup needs to build with the MSVC toolchain, and thus needs to be statically linked to the CRT so that the custom msi actions don't depend on an unavailable CRT dll. https://github.com/rust-lang-nursery/rustup.rs/issues/660
• It looks to me like the CI still needs to be modified to generate a new rustup-msi.exe.sha256 file for upgrades
• appveyor.yml artifacts need to be adjusted for the msi case
• need a strategy for having old non-msi rustup upgrade to the msi rustup. Before we're ready to deploy we can probably put the code in place and have it activated through an environment variable, so that we can test without disrupting existing users.

@Boddlnagg can you give a braindump of other remaining work?

[Boddlnagg]

I had already prepared a table of steps that need to be executed in the final installer (this should probably also be placed somewhere in the code as documentation). I marked those that are already implemented (unfortunately checkboxes don't work within tables). Some steps also need a bit of discussion (e.g. uninstall previous MSI-installed Rust automatically? Is cleanup_legacy() still required?).

The biggest chunk, where I could really use some help, is "install default toolchain, show status/progress in UI". This requires a refactoring of the status reporting during toolchain installation.

Condition	Responsibility	Description
Upgrade	rustup.exe	rustup self upgrade checks if update is available and runs msiexec /i https://path/to/rustup.msi
Uninstall	rustup.exe	[DONE] rustup self uninstall fetches installed product code from registry and runs msiexec /x {<product-code>}
Install	MSI/WiX	Don't allow installation if using Windows < 7 (?)
Install	MSI/WiX	Check if Rust MSI is installed and uninstall automatically (show warning at least)?
Any (?)	MSI/WiX	[DONE] Installer runs RustupPrepare CA:
Install	RustupPrepare (CA) or MSI/WiX	Maybe run do_pre_install_sanity_checks?
Any (?)	RustupPrepare (CA)	Set EXISTS := rustup installed? (check if .multirust and/or .cargo\bin\rustup.exe exist)
Any (?)	RustupPrepare (CA)	Get installation path using utils::cargo_home() (currently using %USERPROFILE%\.rustup-test instead)
Install	MSI/WiX	Show UI for selecting custom installation options (Add to PATH? Default toolchain?)
Install	SetInstallOptions (CA/WiX)	[DONE] Pass Installation options to deferred CA
Upgrade	MSI/WiX	[DONE] Automatically run RemoveExistingProducts during upgrade
Uninstall/Upgrade	MSI/WiX	[DONE] Remove bin directory from PATH (automatically scheduled before RemoveFiles)
Uninstall/Upgrade	MSI/WiX	[DONE] RemoveFiles removes rustup.exe
Uninstall	MSI/WiX	[DONE] Run RustupUninstall CA (after RemoveFiles), only for true Uninstall (not for Upgrade)
Uninstall	RustupUninstall (CA)	Remove .cargo and .multirust, using utils::remove_dir (currently deleting .rustup-test instead)
Install/Upgrade	MSI/WiX	[DONE] InstallFiles extracts and installs rustup.exe
Install/Upgrade	MSI/WiX	[DONE] Run RustupInstall CA:
Install, if EXISTS	RustupInstall (CA)	Remove from PATH (old method, if pre-MSI version is installed)
Install, if EXISTS	RustupInstall (CA)	Maybe run cleanup_legacy()
Install/Upgrade	RustupInstall (CA)	[DONE] Create hardlinks for other bins
Install, if NOT EXISTS	RustupInstall (CA)	(If .multirust does not exist – does it matter if it does?) install default toolchain, show status/progress in UI
Install/Upgrade	MSI/WiX	[DONE] Add bin directory to PATH

CA is short for "Custom Action". "RustupPrepare" is currently called "RustupSetInstallLocation" but should be renamed.

[Boddlnagg]

Also, most of the rustup self update and rustup self uninstall tests won't work anymore, because rustup only runs msiexec and exits. The tests can only check the results once msiexec has finished. So maybe new tests need to be written that explicitly target the MSI (if it is even possible to test MSIs).

[cyplo]

Great progress, thank you !

[Boddlnagg]

I tried to improve the error handling inside the Custom Actions, but found that Custom Actions always should have a corresponding Rollback Custom Action that rolls back the changes done by the normal CA whenever one of the installation steps fails. There is an option to disable rollback completely, but this doesn't seem to work reliably (in my tests, a rollback was performed regardless). All the possible scenarios (successfull/failed installation/upgrade/uninstall) need to be tested thoroughly, so to do it right, everyone recommends to use Custom Actions as sparingly as possible, and instead rely on proven built-in actions. We won't be able to get rid of all CAs (e.g. to create hardlinks and do the toolchain install), but we should wherever we can, even though this means less shared code between platforms. I don't know how important it is that rollback works correctly, but MSI users usually expect it to work (on a failed installation the installer even explicitly says "Your system has not been modified").