Closed colemickens closed 6 years ago
colemickens
commented
6 years ago cc: @copumpkin I think you refactored some stuff in make-disk-image.nix, probably since the last time anyone tried to build an Azure image. Do you have any pointers?
cc: @jbgi I saw you make an azure-image PR, have you seen this? What branch of nixpkgs do you build with?
colemickens
commented
6 years ago It seems that switch_root hangs? I added a bit of set -x to the vm-run-stage{1,2} scripts:
...
+ echo 'mounting Nix store...'
mounting Nix store...
+ mkdir -p /fs/nix/store
+ mount -t 9p store /fs/nix/store -o 'trans=virtio,version=9p2000.L,cache=loose'
+ mkdir -p /fs/tmp /fs/run /fs/var
+ mount -t tmpfs -o 'mode=1777' none /fs/tmp
+ mount -t tmpfs -o 'mode=755' none /fs/run
+ ln -sfn /run /fs/var/run
+ echo 'mounting host'"'"'s temporary directory...'
mounting host's temporary directory...
+ mkdir -p /fs/tmp/xchg
+ mount -t 9p xchg /fs/tmp/xchg -o 'trans=virtio,version=9p2000.L,cache=loose'
+ mkdir -p /fs/proc
+ mount -t proc none /fs/proc
+ mkdir -p /fs/sys
+ mount -t sysfs none /fs/sys
+ mkdir -p /fs/etc
+ ln -sf /proc/mounts /fs/etc/mtab
+ echo '127.0.0.1 localhost'
+ echo 'starting stage 2 (/nix/store/w4gk2prvisgmg4f0vr69ci8adxbwl1h0-vm-run-stage2)'
starting stage 2 (/nix/store/w4gk2prvisgmg4f0vr69ci8adxbwl1h0-vm-run-stage2)
+ exec switch_root /fs /nix/store/w4gk2prvisgmg4f0vr69ci8adxbwl1h0-vm-run-stage2 /nix/store/1z347yrqzf9kj00gp553pdyafx5p940p-azure-image
[hangs]
copumpkin
commented
6 years ago I think @edolstra basically rewrote most of the image building code since that refactor
colemickens
commented
6 years ago It looks like is 9pfs related. A similar thread: https://lists.gt.net/linux/kernel/2599597
jbgi
commented
6 years ago @colemickens I did not try to build an image. I originally just used the existing 16.09 image, upgraded it from time to get the 18.03-based image we use now.
colemickens
commented
6 years ago @jbgi thanks for the information.
It seems that I can build images on one of my local machines. Must be an oddity of running qemu in an Azure VM or something?
I don't know whether to leave this open or closed. I'll leave it up to someone else.
dezgeg
commented
6 years ago @colemickens I think all my patches about 9pfs hangs are merged in recent kernels, that shouldn't be the problem.
dezgeg
commented
6 years ago It seems that I can build images on one of my local machines. Must be an oddity of running qemu in an Azure VM or something?
That's almost certainly it. Nested hardware accelerated virtualization seems just to be horribly broken in Linux.
colemickens
commented
6 years ago This had worked in the past, but I think the VM I was building on yesterday is one of the SKUs that specifically has nested virtualization enabled.
I may end up patching the qemu invocation to disable KVM. It will probably be net faster based on how slow my upload is. Thanks for mentioning it, I hadn't really thought about it.
copumpkin
commented
6 years ago Yeah, I've also found that nested virtualization has basically stopped working entirely for me in the past year or so.
colemickens
commented
6 years ago I think this actually came down to me using a commit from nixos-unstable-small. I had been building images for Azure for a few weeks on an Azure VM without issue, and without having to take steps to disable nested virtualization. Must've been something weird with qemu/kvm/etc at a single point in time... Either way, closing this out.
Issue description
I can't get
azure-imageto build with a branch ofnixos-unstable-small. It hangs atstarting stage 2 (/nix/store/lp6sd4r785qhy77pxvfbyvfhk0cp5739-vm-run-stage2)Steps to reproduce
Output:
FULL OUTPUT: https://gist.github.com/colemickens/ee6030926248ee2d0d305c4fd120506d
Technical details
(the above should cover it)
Please run
nix-shell -p nix-info --run "nix-info -m"and paste the results.(I'm not sure this will be applicable since the
./create-azurescript references it's own nixpkgs enlistment via a local path...)