Alongside OS & Refresh Install Options

[mmstick]

Requires https://github.com/pop-os/distinst/pull/132

Refresh Install

This takes Cassidy's work with the DecryptionDialog at https://github.com/elementary/installer/pull/333, and integrates it with distinst, as well as a new RefreshView to list detected refresh options and to make them happen. This will require a few changes in distinst to expose some previously-hidden APIs, and to fix some issues with installing (updating the recovery partition fails due to running out of space).

Screenshots

Testing

Perform a clean install with and without encryption, and then do the following:

• Create a few user accounts, with some files
• Reboot and use the refresh install option the second time around
• Validate that the files are there, and that the user is still part of the sudo group

For the UI, these can be tested:

• Typing the wrong password / duplicate PV in the decryption menu should display an error dialog
• When only one refresh option is available, the UI should immediately progress to installing it
• When multiple options are possible, there's a UI that will let the user select which one to choose.
• If no LUKS devices are discovered, then the decryption dialog bar on the "Try Install" screen should be hidden by default.
• After decrypting all devices that can be decrypted, the dialog bar should be hidden.
• If multiple devices were found, and a device has been decrypted, an unlock icon should be on that partition's entry in the decryption dialog, instead of a locked icon.

Alongside OS Install

The work had already been done in distinst, so it was just a matter of exposing it in the UI. By sharing UI code with the Refresh install view, both views have the same option selection view, but the alongside choice will also feature a slider for determining how much a partition should be shrunk to make room for the new install.

Screenshots

Sector Slider

Testing

• Perform an unencrypted install, and then attempting to perform an unencrypted and/or encrypted install next to it.
• Attempt to install alongside Windows
• Make a large region of free space, next to an install, and install to that.

[jackpot51]

@mmstick Please fix the build issues

[mmstick]

@jackpot51 It should build once https://github.com/pop-os/distinst/pull/132 is merged, since it requires some FFI and script changes.

[WatchMkr]

Why would a user want to keep the backup install? What would they do with it?

[mmstick]

@WatchMkr They may:

• have configurations that they'd like to keep and/or compare between the previous install and the new install
• Be able to recover the previous install if they find problems with the new install

[WatchMkr]

Few things:

Let's move the "Applications installed on the system...." copy to the previous page "Refresh Install" description. To shorten it up we can use "Reinstall while keeping user accounts and files. Applications will need to be reinstalled manually." This way we can remove the extra step if there's only one possible target.

Since the drive is unlocked in the third screenshot, should the unlock dialog be removed?

Regarding prompting to keep the old install, I don't like this option for a few reasons. -- As a user, the affirmative action is to choose the green button; however, I have no utility to restore to this install. I don't know why I should choose yes or no. My instinct is to keep the backup just-in-case but that's likely unnecessary and a poor default choice. -- It breaks with the application UI (fixable) -- I don't know how much space it will take. (fixable) -- We're treating it somewhat like a snapshot backup but not considering other methods and technologies that might do this better. I'd prefer to consider snapshots and backups outside of the installer before thinking through how it affects the installer.

[mmstick]

@WatchMkr

Since the drive is unlocked in the third screenshot, should the unlock dialog be removed?

Yes, I'm working on the means to do that.

I do like the option to keep it, myself, but I'll remove it for now.

[mmstick]

I think this is ready for some testing. This will require to be tested alongside https://github.com/pop-os/distinst/pull/132, which will need basically every part of the installer to be tested again.

As soon as this is merged, I'll be adding the multiboot install option to the UI. An upgrade option could also be added to the UI somehow, since the installer can now compare the name and version of the OS to install with the name and version of the OS being refreshed.

[mmstick]

Found an issue with secondary groups. Will have that fixed soon.

[mmstick]

The "Install Alongside OS" option is ready now. It will list up to two options per block device: installing to the largest free region that meets minimum requiements, and shrinking the partition with the largest unused space.

[brs17]

The first thing I noticed with this pr is the need to scroll to see all of the installer options. I think that makes the installer feel heavier and is probably not great UX.

[brs17]

The first thing I tried to do was do a "Refresh Install". The install I had on the disk was a custom partitioned drive (efi, /swap, /, /home, /tmp). After selecting this option, the installer immediately failed:

I then selected to try installing again. This time when selecting "Refresh Install", I got the next page.

Until I noticed that the button to proceed was enabled. See the following screenshot:

It seems as though something is missing here (as it differs from the screenshot you provided).

Proceeding here, this fails as well

[brs17]

With the previous having failed, I then attempted to "Install alongside OS".

This seems to be quite text heavy and potentially confusing in what is going to happen exactly.

Once selecting one of the options, the selection itself looks a little odd, with the orange highlight so far to the left of the actual option (since the button is that wide).

Also I am not really sure here what partitions those options are. (which one is partition 4?)

My confusion continues on the next page:

I think this could be shown better to explain what is happening to both partitions.

Moving the slider all the way to the left, I have a non-zero value. Why is that? Is MiB the best unit to use here with >9000MiB being the value I am looking at? With custom partitioning we just use gparted for all of the partitioning work, why not continue that here (and benefit from the visual contexts it provides)?

[mmstick]

Seems the recent distinst changes have broken the options. Probably the PARTUUID change.

Moving the slider all the way to the left, I have a non-zero value.

The minimum value for the slider is the minimum required size for an install. There has to be enough room for all the partitions that distinst will create in the free space of that shrunk partition.

With custom partitioning we just use gparted for all of the partitioning work, why not continue that here (and benefit from the visual contexts it provides)?

Main reason is that they don't need to. The installer will automate that process for them, whether it's choosing to install to unpartitioned space on the disk, or shrinking an existing partition to make room for the new install.

[WatchMkr]

Try or Install page UX Changes:

Change the page title from "Try or Install" to "Install" Change "Try Demo Mode" to a button in the ancillary action location at the bottom left of the window.

Copy change: Install Alongside another Operating System Add Pop!_OS next to one or more existing OS installations.

Alignment: Please align the Refresh Install option title and description with the other options.

I'll take a peak at the refresh and install alongside OS options next.

[brs17]

Looking at build: 9aaae3c after doing a full disk encrypted install:

@WatchMkr, when you suggested moving Try Demo Mode to the bottom left, did you mean the following way or making it a button like the Next and Back button?

[WatchMkr]

A button like the Back and Next buttons.

[brs17]

When selecting to Unlock Encrypted Storage a popup appears:

The only other time a popup appears is when gparted opens. Do we want a popup here? I could follow either argument.

I did notice, one has to press ESC twice to close the popup which seems odd.

Unlock page:

Wrong password page:

Thoughts @WatchMkr @jackpot51 ?

[brs17]

The minimum value for the slider is the minimum required size for an install. There has to be enough room for all the partitions that distinst will create in the free space of that shrunk partition.

With custom partitioning we just use gparted for all of the partitioning work, why not continue that here (and benefit from the visual contexts it provides)?

I guess I kinda felt like a lot of this was presented without as good context as I would have liked.

[WatchMkr]

I think it's correct to use a pop-up for encryption unlock. If there is only one possible partition to unlock, skip selection and go directly to the second window. The copy for the second window should be:

"Unlock partition /dev/$partname" along with the Password and Device Name entry boxes. Is Device Name already filled in?

[brs17]

Device name is already filled in.

I also saw that when typing in an incorrect password, once I pressed back, the incorrect password was still in the field, I imagine that should be cleared as well.

[brs17]

I don't know why I didn't think about it sooner but I should probably use Peek or something and just record myself going through the process.

[evertiro]

Looking at build: 9aaae3c after doing a full disk encrypted install:

@WatchMkr, when you suggested moving Try Demo Mode to the bottom left, did you mean the following way or making it a button like the Next and Back button?

Having Install, Refresh, Custom, Alongside, Demo... it gets bulky fast. Is there any good reason we can't do what Ubuntu did and have the first dialog simply be Try/Install, with the installation options being presented only after the user has indicated they want to install?

That would allow us to have a bit of a clear divide between "try" and "install", while still keeping the initial install screen from feeling overly cluttered.

[cassidyjames]

@evertiro you can check the full design spec at https://github.com/elementary/installer/wiki/Try-or-Install. But essentially the reasoning is that adding another step before we can begin installing just slows the user down. We should be able to offer a few sane options here without it getting too cluttered up.

[WatchMkr]

@evertiro we'd like to continue the user down the most likely and preferred path. Installing being the primary purpose and using demo mode or the live session being ancillary.

[WatchMkr]

The Install Alongside feature needs some design direction. There was work done already but I can't find it. @cassidyjames do you have the original work on that? I don't see it on the wiki.

[evertiro]

Oh I 100% agree that things should be streamlined and as simple as possible for users. Unfortunately, as the screenshot posted by @brs17 demonstrates, that particular screen is far from simple... It's downright cluttered! I'm not saying that going with a completely separate window is the right solution, only that there needs to be some sort of division between "try" and "install" for the users who aren't already familiar with Linux.

[brs17]

I understand your concern, let's take a look once the "try" button becomes a button like the "next" button and reaccess once we see screenshots.

EDIT: I think moving the Try button is what we need to clean it up...note not all the options appear all the time (they are dependent on what is already on the system)

[evertiro]

EDIT: I think moving the Try button is what we need to clean it up...note not all the options appear all the time (they are dependent on what is already on the system)

This may be true. I'd like to see what you're envisioning for the Try button though.

[cassidyjames]

@WatchMkr there wasn't much set in stone. I was actually working on this today in the elementary Slack. The original (i.e. really old) mockup looked like this:

I'm working on something (VERY in-progress) that looks like this:

[cassidyjames]

@WatchMkr or more Pop! styled:

I have a PR open mostly for a prototype UI at https://github.com/elementary/installer/pull/365

[WatchMkr]

That looks great. What if there are more than one OS’s installed?

[cassidyjames]

@WatchMkr the intent was that that's a different use case; a single other OS is probably common, even for a less advanced user, so we handle that case here. But having multiple OSes likely signals that the user is more comfortable with setting up their installs how they want, so Custom should still be sufficient.

The different scenarios are listed out at https://github.com/elementary/installer/wiki/Try-or-Install

[WatchMkr]

Sounds good.

[mmstick]

@WatchMkr That should work regardless of how many operating systems are installed, though the word would change depending on what is being shrunk. Possible install locations are:

• The largest unused region on a disk, in which case no resizing of an existing OS is required.
• An existing partition that has on OS on it, in which case we are shrinking that OS's partition.
• An existing partition that does not have an OS on it, in which case we're just shrinking a data partition.

[mmstick]

I've fixed the refresh installs. PR coming soon for distinst.

[mmstick]

This is what the install view looks like now, with all the options in place.

[evertiro]

This is what the install view looks like now, with all the options in place.

That looks a TON better! Though personally, I think the demo button is a bit understated.

[brs17]

@cassidyjames: @WatchMkr, @mmstick, and I just sat down to discuss these new additions to the installer, we like the designs you've put up thus far (and vids we've seen in Slack).

@mmstick and @cassidyjames might be worth being in touch as you two make progress so we aren't reinventing the wheel :stuck_out_tongue:

[mmstick]

WIP changes to the views

[mmstick]

@brs17 This is ready for testing, though you'll have to compile this from source with the refresh-fixes branch of distinst installed.

[brs17]

@mmstick why does it need to be compiled from source?

[mmstick]

@brs17 Because this won't build without the refresh-fixes version of distinst.

[brs17]

Well then let's get https://github.com/pop-os/distinst/pull/165 in there as well.

[mmstick]

Now with OS logos:

Once we get an icon for Pop in our icon theme, that can be displayed as well[0].

[0] https://github.com/pop-os/icon-theme/issues/39

[mmstick]

[brs17]

Testing

Template

• [ ] [new install + old install]

---

Intel/AMD

• [x] Install Pop 18.04 unencrypted
• [x] 18.04 unencrypted Refresh install
• [x] Pop 18.10 + Pop 18.04 unencrypted
• [x] Refresh 18.10 in dualboot
• [x] Install Pop 18.10 encrypted
• [x] 18.10 encrypted Refresh install
• [x] 18.10 unencrypted
• [x] Pop 18.04 + Pop 18.10 unencrypted
• [x] Refresh 18.04 in dualboot
• [x] Pop 18.04 + Ubuntu 18.04(systemdboot + grub)
• [x] Refresh 18.04 in dualboot
• [ ] Pop 18.04 + Fedora 29 + Ubuntu 18.04 + Windows (Windows on separate drive)
• [ ] Pop 18.04 + Windows
• [ ] Pop 18.04 + Fedora on LVM
• [ ] Pop 18.10 + Ubuntu 18.04(systemdboot + grub)
• [ ] Refresh 18.10 in dualboot
• [ ] Pop 18.10 + Fedora 29 + Ubuntu 18.04 + Windows (Windows on separate drive)
• [ ] Pop 18.10 + Windows
• [ ] Pop 18.10 + Fedora on LVM

---

NVIDIA

• Install Pop 18.04 encrypted
• [ ] 18.04 encrypted Refresh install
• [ ] Pop 18.10 + Pop 18.04 encrypted
• [ ] Refresh 18.10 in dualboot
• Install Pop 18.10 encrypted
• [ ] 18.10 encrypted Refresh install
• [ ] Pop 18.04 + Pop 18.10 encrypted
• [ ] Refresh 18.04 in dualboot
• [ ] Pop 18.04 + Ubuntu 18.04(systemdboot + grub)
• [ ] Refresh 18.04 in dualboot
• [ ] Pop 18.04 + Fedora 29 + Ubuntu 18.04 + Windows (Windows on separate drive)
• [ ] Pop 18.04 + Windows
• [ ] Pop 18.04 + Fedora on LVM
• [ ] Pop 18.10 + Ubuntu 18.04(systemdboot + grub)
• [ ] Refresh 18.10 in dualboot
• [ ] Pop 18.10 + Fedora 29 + Ubuntu 18.04 + Windows (Windows on separate drive)
• [ ] Pop 18.10 + Windows
• [ ] Pop 18.10 + Fedora on LVM

---

• [ ] Pop 18.04 + Pop 18.04/18.10 on different drive
• [ ] Pop 18.04 + empty partition
• [ ] Pop 18.04 + empty drive
• [ ] Pop 18.10 + Pop 18.04/18.10 on different drive
• [ ] Pop 18.10 + empty partition
• [ ] Pop 18.10 + empty drive

Testing "Refresh" Install

• [ ] Test a Refresh install of 18.04 with an encrypted home folder
• [ ] Test a Refresh install of 18.10 with an encrypted home folder
• [ ] 18.04 unencrypted
• [ ] 18.10 unencrypted

[brs17]

• [ ] "Refresh" option was available when installing two different operating systems (Pop and Ubuntu, Pop 18.10 with 18.04)

~- [ ] In 18.04 the "refresh" icon is not very visible: EDIT: Fixed for 18.10 here with https://github.com/pop-os/gtk-theme/pull/248. Still an issue in 18.04.~

• [ ] Alongside OS option is not visible when attempting to install Pop!_OS 18.10 alongside an encrypted Pop!_OS 18.04 install (or the other way around). The only option shown was a "Refresh install" which shouldn't be an option in this case. I was able to do an "alongside" install after doing a custom install of the operating system.

• [ ] On the "Resize OS" page, the left side of the installer ("Resize OS" text as well as Zoe image) is smaller than the left side content is on other pages.

• [ ] When doing an "Alongside" install, The max size to shrink the current install is ~30.3GB (20GB of free space). The minimum size of the new install was 14.3GB (0 bytes free). Why shouldn't both sides require free space?

• [x] When successfully doing an "Alongside OS" install, two efi boot partitions are created and the user must switch OSes from the bios boot menu. Do we want this to be the case, or should the two operating systems share a boot parititon?

• [x] "Refresh" install failed from recovery partition. From an external drive the "Refresh" install worked successfully.

• [x] The password field presented when decrypting storage does not put focus on the password entry field (does not focus on any text box).

• [ ] When looking at options available with Ubuntu already installed, when selecting "Alongside" or "Custom", the padding around the "Try Demo Mode" button changes (increases). Padding should be consistent.

• [ ] Installer logs for when refresh install failed of 18.10 on dualboot of 18.04 and 18.10: installer.log

• [ ] I noticed on a refresh install on encrypted systems the location for possible refresh installs are listed similar to: /dev/mapper/data-root whereas if it is an unencrypted system, listed as: /dev/sdx or /dev/nvmexny I would think it might be more clear/easier to recognize an install from /dev/sdx rather than /dev/mapper/data-root. Is there anything that could be done here?

[brs17]

• Just tried an "Alongside" install of Pop!_OS 18.04 with Pop!_OS 18.04 and got this failure: installer2.log

[mmstick]

@brs17

On the "Resize OS" page, the left side of the installer ("Resize OS" text as well as Zoe image) is smaller than the left side content is on other pages.

Fixed by removing the elementary margins (48) from the content grid.

The password field presented when decrypting storage does not put focus on the password entry field (does not focus on any text box).

Fixed the focus for the select and unlock buttons, as well as their sensitivities. Also added some constraints to the PV name entry so that users won't be able to input invalid characters.

When looking at options available with Ubuntu already installed, when selecting "Alongside" or "Custom", the padding around the "Try Demo Mode" button changes (increases). Padding should be consistent.

Had the same issue as the partitioning view. That's fixed now.

Just tried an "Alongside" install of Pop!_OS 18.04 with Pop!_OS 18.04 and got this failure: installer2.log

This only happens with 18.04? For whatever reason, fsck failed with "Operation not permitted".

Installer logs for when refresh install failed of 18.10 on dualboot of 18.04 and 18.10: installer.log

Haven't been able to replicate so far. Will keep testing. I've added more context to the error messages in that region of distinst so that future logs will have a bit more detail on where this happened, exactly.

The minimum size of the new install was 14.3GB (0 bytes free)

Do you think 5GB of overhead for a new install would be enough?

I would think it might be more clear/easier to recognize an install from /dev/sdx rather than /dev/mapper/data-root. Is there anything that could be done here?

It's possible to resolve a LV to a VG, then to a PV, and then back to the original physical block device. I could add that behavior here.

When successfully doing an "Alongside OS" install, two efi boot partitions are created and the user must switch OSes from the bios boot menu. Do we want this to be the case, or should the two operating systems share a boot parititon?

Supporting this just involves copying the contents of the other EFI partitions to the new EFI partition, if it is safe to do so. I did experiment with installing to an existing EFI partition. Would need to add a check to determine whether it's possible to re-use an existing EFI partition (free space + overhead).

We might also want to increase the default size of the EFI partition to 1G if we're supporting multiple operating systems + older kernels on the same EFI partition as Pop. Pop by itself uses about 50% of the 500M EFI partition, which leaves enough room for Windows (100M), but would not be enough for a dual boot of Pop or another Linux OS.

[mmstick]

The help dialog should no longer open multiple instances now. GParted will also no longer open more than once if the modify button is clicked multiple times in advance.