Upgrading the latest bootable Azure image to stable requires stepping, results in broken image

[colemickens]

Issue description

1. The Azure images are old. One of them doesn't work. I believe the one that does work is the NixOps bootstrap image and lacks a channel out of the box. For the sake of this bug, I'm starting with this VHD: "https://nixos.blob.core.windows.net/images/nixos-image-16.09.1694.019dcc3-x86_64-linux.vhd"

2. You cannot upgrade from the latest to 19.03 or nixos-unstable directly. Instead I've found that I can do 16.09 -> 17.03 -> 18.03 -> 19.03 and everything works out. (I'll include the outputs from failed attempts when I try again. It did try to workaround not having the right version of Nix, but it basically kept failing on an evaluation in some specific gtk derivation deep down in.)

3. Except not really. If you upgrade to nixos-unstable and shut down the VM and try to start it, it will fail. It looks like the Grub entries are using FSUUID values and maybe the right kernel module isn't loaded for them to appear under those names? All I know is that the partuuid is there before shutdown, and then grub complains it can't find it on boot.

I'll be repeating this again today, in an attempt to get something work stably.

[matthewbauer]

I thought you could go from 17.09 -> 19.03 without any issue, but I think I only tried it once. Going forward, I'd like to keep it so you can go from 17.09 to any future version. But it's pretty difficult to backport those things pre-17.09. Maybe we can make some tests to verify this works?

Anyway, could we just upload a newer azure image to fix this? I'm not sure who has permissions to do that ? Someone must still have it... /cc @rbvermaa @edolstra @vcunat @Phreedom

related to #3986 and #36262

[vcunat]

I do not have it and do not know anyone who does.

[colemickens]

The upgrade to 17.09 takes, the upgrade from there to 19.03 does not.

$ nix-channel --add https://nixos.org/channels/nixos-17.09 nixos; nix-channel --update; nixos-rebuild switch

$ nix-channel --add https://nixos.org/channels/nixos-19.03 nixos; nix-channel --update; nixos-rebuild switch &> ~/log; cat ~/log | curl -F 'f:1=<-' ix.io
...
http://ix.io/1KMH

I've copied that log below. In this time around, I used 16.09->17.09->18.09->19.03, since I wanted to test the 17.09->19.03 leg specifically, then resorted to 17.09->18.09->19.03.

``` error: evaluation aborted with the following error message: ‘ This version of Nixpkgs requires Nix >= 2.0, please upgrade: - If you are running NixOS, `nixos-rebuild' can be used to upgrade your system. - Alternatively, with Nix > 2.0 `nix upgrade-nix' can be used to imperatively upgrade Nix. You may use `nix-env --version' to check which version you have. - If you installed Nix using the install script (https://nixos.org/nix/install), it is safe to upgrade by running it again: curl https://nixos.org/nix/install | sh For more information, please see the NixOS release notes at https://nixos.org/nixos/manual or locally at /nix/store/mv6wzf1j7g0avqcxn6ggwbldax3yqkns-nixos-19.03.172786.606306e0eaa/nixos/nixos/doc/manual/release-notes. If you need further help, see https://nixos.org/nixos/support.html ’ (use ‘--show-trace’ to show detailed location information) building Nix... error: evaluation aborted with the following error message: ‘ This version of Nixpkgs requires Nix >= 2.0, please upgrade: - If you are running NixOS, `nixos-rebuild' can be used to upgrade your system. - Alternatively, with Nix > 2.0 `nix upgrade-nix' can be used to imperatively upgrade Nix. You may use `nix-env --version' to check which version you have. - If you installed Nix using the install script (https://nixos.org/nix/install), it is safe to upgrade by running it again: curl https://nixos.org/nix/install | sh For more information, please see the NixOS release notes at https://nixos.org/nixos/manual or locally at /nix/store/mv6wzf1j7g0avqcxn6ggwbldax3yqkns-nixos-19.03.172786.606306e0eaa/nixos/nixos/doc/manual/release-notes. If you need further help, see https://nixos.org/nixos/support.html ’ (use ‘--show-trace’ to show detailed location information) error: evaluation aborted with the following error message: ‘ This version of Nixpkgs requires Nix >= 2.0, please upgrade: - If you are running NixOS, `nixos-rebuild' can be used to upgrade your system. - Alternatively, with Nix > 2.0 `nix upgrade-nix' can be used to imperatively upgrade Nix. You may use `nix-env --version' to check which version you have. - If you installed Nix using the install script (https://nixos.org/nix/install), it is safe to upgrade by running it again: curl https://nixos.org/nix/install | sh For more information, please see the NixOS release notes at https://nixos.org/nixos/manual or locally at /nix/store/mv6wzf1j7g0avqcxn6ggwbldax3yqkns-nixos-19.03.172786.606306e0eaa/nixos/nixos/doc/manual/release-notes. If you need further help, see https://nixos.org/nixos/support.html ’ /tmp/nixos-rebuild.mIgdnA/nix building the system configuration... error: evaluation aborted with the following error message: ‘ This version of Nixpkgs requires Nix >= 2.0, please upgrade: - If you are running NixOS, `nixos-rebuild' can be used to upgrade your system. - Alternatively, with Nix > 2.0 `nix upgrade-nix' can be used to imperatively upgrade Nix. You may use `nix-env --version' to check which version you have. - If you installed Nix using the install script (https://nixos.org/nix/install), it is safe to upgrade by running it again: curl https://nixos.org/nix/install | sh For more information, please see the NixOS release notes at https://nixos.org/nixos/manual or locally at /nix/store/mv6wzf1j7g0avqcxn6ggwbldax3yqkns-nixos-19.03.172786.606306e0eaa/nixos/nixos/doc/manual/release-notes. If you need further help, see https://nixos.org/nixos/support.html ’ (use ‘--show-trace’ to show detailed location information) ```

[colemickens]

(That log isn't the most interesting one, but I don't want to try more combos right now unless someone really wants me to find the jump that causes the weird GTK eval error.)

On to the boot issue. Directly after upgrading all of the way to 19.03...

sh-4.3# ls -al  /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 100 May 30 14:15 .
drwxr-xr-x 8 root root 160 May 30 14:15 ..
lrwxrwxrwx 1 root root  10 Jun  3 05:37 3506ce47-f82c-438c-bed2-d91bd2c3bd72 -> ../../sdc1
lrwxrwxrwx 1 root root  10 Jun  3 05:37 b3e8f4c6-64f2-4f13-8eb1-a9d5e83279e3 -> ../../sda1
lrwxrwxrwx 1 root root  10 Jun  3 05:37 ba77ec3b-9545-46c3-857f-5070a8cbc2d4 -> ../../sdb1

sh-4.3# cat /boot/grub/grub.cfg

``` # Automatically generated. DO NOT EDIT THIS FILE! search --set=drive2 --fs-uuid 3506ce47-f82c-438c-bed2-d91bd2c3bd72 search --set=drive1 --fs-uuid 3506ce47-f82c-438c-bed2-d91bd2c3bd72 if [ -s $prefix/grubenv ]; then load_env fi # ‘grub-reboot’ sets a one-time saved entry, which we process here and # then delete. if [ "${next_entry}" ]; then set default="${next_entry}" set next_entry= save_env next_entry set timeout=1 else set default=0 set timeout=0 fi # Setup the graphics stack for bios and efi systems if [ "${grub_platform}" = "efi" ]; then insmod efi_gop insmod efi_uga else insmod vbe fi insmod font if loadfont ($drive1)/boot/converted-font.pf2; then insmod gfxterm if [ "${grub_platform}" = "efi" ]; then set gfxmode=auto set gfxpayload=keep else set gfxmode=1024x768 set gfxpayload=text fi terminal_output gfxterm fi menuentry "NixOS - Default" { search --set=drive1 --fs-uuid 3506ce47-f82c-438c-bed2-d91bd2c3bd72 search --set=drive2 --fs-uuid 3506ce47-f82c-438c-bed2-d91bd2c3bd72 linux ($drive2)/nix/store/cbmq4jbh8m5xr9dl2s1dnnpf95brcwpd-linux-4.19.47/bzImage systemConfig=/nix/store/fs5qsfn7mlm2wbzzrc2lc32y2a6cmlfy-nixos-system-unnamed-19.03.172786.606306e0eaa init=/nix/store/fs5qsfn7mlm2wbzzrc2lc32y2a6cmlfy-nixos-system-unnamed-19.03.172786.606306e0eaa/init panic=1 boot.panic_on_fail console=ttyS0 earlyprintk=ttyS0 rootdelay=300 panic=1 boot.panic_on_fail loglevel=4 initrd ($drive2)/nix/store/cvk7d0yr3pccvzh247l4zi0y0ydn3hly-initrd/initrd } submenu "NixOS - All configurations" { } ```

Results in:

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.