search

clalancette / oz

Automated installation for guest images
GNU Lesser General Public License v2.1
310 stars 129 forks source link

[git7920f12] oz-install hangs while trying to create the disk image file #98

Closed kashyapc closed 10 years ago

kashyapc commented 11 years ago

Built from git:

$ rpm -q oz
oz-0.11.0-0.20130711113852git7920f12.fc19.noarch
$ git log | head -1
commit 7920f121cfe960265ac9e2ed2a4563764b1ccd97

That's the script I used to create an Oz guest -- https://github.com/kashyapc/virt-scripts/blob/master/oz/stdout-examples/f19/oz-create-f19.bash

With this TDL:

<template>
  <name>f19min</name>
  <os>
    <name>Fedora</name>
    <version>19</version>
    <arch>x86_64</arch>
    <install type='url'>
      <url>http://download.foo.bar.com/pub/fedora/linux/releases/19/Fedora/x86_64/os</url>
    </install>
    <rootpw>redhat</rootpw>
  </os>
  <description>Fedora 19</description>
  <disk>
    <size>3</size>
  </disk>

Here's where it just hangs:

$ ./oz-jeos.bash f19min f19                                                                                                                                                  
1
IP forwarding enabled
DEBUG:oz.Guest.FedoraGuest:libvirt bridge name is virbr0
DEBUG:oz.Guest.FedoraGuest:Libvirt type is kvm
DEBUG:oz.Guest.FedoraGuest:Name: f19min, UUID: ea3cf72c-73b5-4cad-bec5-c325411587c8
DEBUG:oz.Guest.FedoraGuest:MAC: 52:54:00:31:97:b8, distro: Fedora
DEBUG:oz.Guest.FedoraGuest:update: 19, arch: x86_64, diskimage: /var/lib/libvirt/images/f19min.dsk
DEBUG:oz.Guest.FedoraGuest:nicmodel: virtio, clockoffset: utc
DEBUG:oz.Guest.FedoraGuest:mousetype: ps2, disk_bus: virtio, disk_dev: vda
DEBUG:oz.Guest.FedoraGuest:icicletmp: /var/lib/oz/icicletmp/f19min, listen_port: 48439
DEBUG:oz.Guest.FedoraGuest:Original ISO path: /var/lib/oz/isos/Fedora19x86_64-url.iso
DEBUG:oz.Guest.FedoraGuest:Modified ISO cache: /var/lib/oz/isos/Fedora19x86_64-url-oz.iso
DEBUG:oz.Guest.FedoraGuest:Output ISO path: /var/lib/libvirt/images/f19min-url-oz.iso
DEBUG:oz.Guest.FedoraGuest:ISO content path: /var/lib/oz/isocontent/f19min-url
INFO:oz.Guest.FedoraGuest:Checking for guest conflicts with f19min
DEBUG:oz.Guest.FedoraGuest:Installtype is URL, trying to do direct kernel boot
DEBUG:oz.Guest.FedoraGuest:Going to write treeinfo to /var/lib/oz/icicletmp/f19min/treeinfo
DEBUG:oz.Guest.FedoraGuest:Trying to get treeinfo from http://download.foo.bar.com/pub/fedora/linux/releases/19/Fedora/x86_64/os//.treeinfo
DEBUG:oz.Guest.FedoraGuest:Got treeinfo, parsing
DEBUG:oz.Guest.FedoraGuest:Returning kernel images/pxeboot/vmlinuz and initrd images/pxeboot/initrd.img
INFO:oz.Guest.FedoraGuest:Fetching the original media
DEBUG:oz.Guest.FedoraGuest:Attempting to get the lock for /var/lib/oz/kernels/Fedora19x86_64-kernel
DEBUG:oz.Guest.FedoraGuest:Got the lock, doing the download
INFO:oz.Guest.FedoraGuest:Original install media available, using cached version
INFO:oz.Guest.FedoraGuest:Fetching the original media
DEBUG:oz.Guest.FedoraGuest:Attempting to get the lock for /var/lib/oz/kernels/Fedora19x86_64-ramdisk
DEBUG:oz.Guest.FedoraGuest:Got the lock, doing the download
INFO:oz.Guest.FedoraGuest:Original install media available, using cached version
DEBUG:oz.Guest.FedoraGuest:Putting the kickstart in place
DEBUG:oz.Guest.FedoraGuest:Writing cpio to /var/lib/oz/icicletmp/f19min/extra.cpio
INFO:oz.Guest.FedoraGuest:Generating 3GB diskimage for f19min

And, there /is/ space on my root partition:

$ df -hT
Filesystem                           Type      Size  Used Avail Use% Mounted on
/dev/mapper/fedora_dhcp207--228-root ext4       63G   51G  9.0G  85% /
devtmpfs                             devtmpfs   12G     0   12G   0% /dev
tmpfs                                tmpfs      12G     0   12G   0% /dev/shm
tmpfs                                tmpfs      12G  676K   12G   1% /run
tmpfs                                tmpfs      12G     0   12G   0% /sys/fs/cgroup
tmpfs                                tmpfs      12G   36K   12G   1% /tmp
/dev/sda1                            ext4      772M  148M  568M  21% /boot
kashyapc commented 11 years ago

Duh, I can't seem to reproduce this on a different machine...

$ cat f19oz.out
DEBUG:oz.Guest.FedoraGuest:libvirt bridge name is virbr0
DEBUG:oz.Guest.FedoraGuest:Libvirt type is kvm
DEBUG:oz.Guest.FedoraGuest:Name: f19oz, UUID: ec73d99c-b81f-43fe-b2ac-ea4e1bb93943
DEBUG:oz.Guest.FedoraGuest:MAC: 52:54:00:8b:aa:de, distro: Fedora
DEBUG:oz.Guest.FedoraGuest:update: 19, arch: x86_64, diskimage: /var/lib/libvirt/images/f19oz.dsk
DEBUG:oz.Guest.FedoraGuest:nicmodel: virtio, clockoffset: utc
DEBUG:oz.Guest.FedoraGuest:mousetype: ps2, disk_bus: virtio, disk_dev: vda
DEBUG:oz.Guest.FedoraGuest:icicletmp: /var/lib/oz/icicletmp/f19oz, listen_port: 14806
DEBUG:oz.Guest.FedoraGuest:Original ISO path: /var/lib/oz/isos/Fedora19x86_64-url.iso
DEBUG:oz.Guest.FedoraGuest:Modified ISO cache: /var/lib/oz/isos/Fedora19x86_64-url-oz.iso
DEBUG:oz.Guest.FedoraGuest:Output ISO path: /var/lib/libvirt/images/f19oz-url-oz.iso
DEBUG:oz.Guest.FedoraGuest:ISO content path: /var/lib/oz/isocontent/f19oz-url
INFO:oz.Guest.FedoraGuest:Checking for guest conflicts with f19oz
DEBUG:oz.Guest.FedoraGuest:Installtype is URL, trying to do direct kernel boot
DEBUG:oz.Guest.FedoraGuest:Going to write treeinfo to /var/lib/oz/icicletmp/f19oz/treeinfo
DEBUG:oz.Guest.FedoraGuest:Trying to get treeinfo from http://download.foo.bar.com/pub/fedora/linux/releases/19/Fedora/x86_64/os//.treeinfo
DEBUG:oz.Guest.FedoraGuest:Got treeinfo, parsing
DEBUG:oz.Guest.FedoraGuest:Returning kernel images/pxeboot/vmlinuz and initrd images/pxeboot/initrd.img
INFO:oz.Guest.FedoraGuest:Fetching the original media
DEBUG:oz.Guest.FedoraGuest:Attempting to get the lock for /var/lib/oz/kernels/Fedora19x86_64-kernel
DEBUG:oz.Guest.FedoraGuest:Got the lock, doing the download
INFO:oz.Guest.FedoraGuest:Original install media available, using cached version
INFO:oz.Guest.FedoraGuest:Fetching the original media
DEBUG:oz.Guest.FedoraGuest:Attempting to get the lock for /var/lib/oz/kernels/Fedora19x86_64-ramdisk
DEBUG:oz.Guest.FedoraGuest:Got the lock, doing the download
INFO:oz.Guest.FedoraGuest:Original install media available, using cached version
DEBUG:oz.Guest.FedoraGuest:Putting the kickstart in place
DEBUG:oz.Guest.FedoraGuest:Writing cpio to /var/lib/oz/icicletmp/f19oz/extra.cpio
INFO:oz.Guest.FedoraGuest:Generating 3GB diskimage for f19oz
INFO:oz.Guest.FedoraGuest:Running install for f19oz
INFO:oz.Guest.FedoraGuest:Generate XML for guest f19oz with bootdev None
DEBUG:oz.Guest.FedoraGuest:Generated XML:
<?xml version="1.0"?>
<domain type="kvm">
  <name>f19oz</name>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <uuid>ec73d99c-b81f-43fe-b2ac-ea4e1bb93943</uuid>
  <clock offset="utc"/>
  <vcpu>1</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type>hvm</type>
    <kernel>/var/lib/libvirt/images/f19oz-kernel</kernel>
    <initrd>/var/lib/libvirt/images/f19oz-ramdisk</initrd>
    <cmdline>method=http://download.foo.bar.com/pub/fedora/linux/releases/19/Fedora/x86_64/os/ ks=file:/ks.cfg</cmdline>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:8b:aa:de"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <serial type="pty">
      <target port="0"/>
    </serial>
    <serial type="tcp">
      <source mode="bind" host="127.0.0.1" service="14806"/>
      <protocol type="raw"/>
      <target port="1"/>
    </serial>
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/var/lib/libvirt/images/f19oz.dsk"/>
      <driver name="qemu" type="raw"/>
    </disk>
  </devices>
</domain>

DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1200/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1190/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1180/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1170/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1160/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1150/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1140/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1130/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1120/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1110/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1100/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1090/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1080/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1070/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1060/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1050/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1040/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to finish installing, 1030/1200
DEBUG:oz.Guest.FedoraGuest:Waiting for f19oz to complete shutdown, 10/10
INFO:oz.Guest.FedoraGuest:Install of f19oz succeeded
INFO:oz.Guest.FedoraGuest:Generate XML for guest f19oz with bootdev hd
DEBUG:oz.Guest.FedoraGuest:Generated XML:
<?xml version="1.0"?>
<domain type="kvm">
  <name>f19oz</name>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <uuid>ec73d99c-b81f-43fe-b2ac-ea4e1bb93943</uuid>
  <clock offset="utc"/>
  <vcpu>1</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:8b:aa:de"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <serial type="pty">
      <target port="0"/>
    </serial>
    <serial type="tcp">
      <source mode="bind" host="127.0.0.1" service="14806"/>
      <protocol type="raw"/>
      <target port="1"/>
    </serial>
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/var/lib/libvirt/images/f19oz.dsk"/>
      <driver name="qemu" type="raw"/>
    </disk>
  </devices>
</domain>

INFO:oz.Guest.FedoraGuest:Cleaning up after install
Libvirt XML was written to f19ozJul_11_2013-18:19:20
clalancette commented 11 years ago

Yeah, and I can't seem to reproduce it here either. It's a bit of an odd bug. The only things I can think to try are:

  1. On the machine with the problem, try to downgrade to Oz 0.10.0 and see if that helps. You can just lie in the TDL and claim that a Fedora 19 installation is a Fedora 17 or 18 one; that should still work. In Oz 0.10, we used truncate() to create the disk images. In Oz 0.11, we are using libvirt to do the same (the benefits are that we can now create qcow2 disk images, etc). So it is possible that somehow libvirt is stuck.
  2. Because of what I mentioned in 1), try restarting libvirt. You can also try checking out the default pools in libvirt (virsh pool-list), and possible deactivating/deleting them.
  3. Run strace when the process and hung and find out where it is hung up. This isn't always easy, but it may be possible to find the call that is hanging, and relate it back to a file descriptor (using lsof), and from there infer what is going on.

Let me know how it goes.

clalancette commented 10 years ago

I'm going to close this out for now. If you run across the problem again, feel free to re-open and we can investigate.