Closed jpeter01 closed 5 years ago
I tried the following to reproduce this issue:
$ git clone https://github.com/SUSE/kiwi-descriptions.git
$ cd kiwi-descriptions/suse/x86_64/suse-leap-15.0-JeOS/
$ sudo kiwi-ng-3 --color-output --type oem system build --target-dir /tmp/mytest/ --description .
and it just builds.
Your target dir --target-dir /srv/kiwi/this_build-4.0.4
what is this ? anything special ? NFS mounted such that rsync might have a problem ? Just guessing here. In any case at the time you think it hangs you can try to repeat the rsync call in an extra shell and see if it outputs something that points to a hang
Nothing special, it is a local xfs filesystem. I tried the rsync manually, it show's that the filesystem is full: ...rsync: mkstemp "/tmp/kiwi_volumes.1h37f1hz/srv/www/squidanalyzer/.sorttable.js.jUvJk0" failed: No space left on device (28) rsync: mkstemp "/tmp/kiwi_volumes.1h37f1hz/srv/www/squidanalyzer/.squidanalyzer.css.0Av67U" failed: No space left on device (28)
I have set the the root partition size to 20G, but during the build process, it is smaller. The root FS size is 4.8 GB during the build process, the final ISO size should be ~1.7 GB.
Additional info:
du -sh /srv/kiwi/huedu-4.0.4/build/image-root/
4,8G /srv/kiwi/huedu-4.0.4/build/image-root/
df -hT
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 2.0G 8.0K 2.0G 1% /dev
tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs tmpfs 2.0G 9.0M 2.0G 1% /run
tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/xvda2 xfs 28G 3.4G 25G 12% /
/dev/mapper/system--disk2--build42-srv xfs 110G 64G 47G 58% /srv
/dev/mapper/vgsystem-LVRoot ext4 6.0G 5.4G 359M 94% /tmp/kiwi_volumes.1h37f1hz
/dev/mapper/vgsystem-home ext4 101M 52M 42M 56% /tmp/kiwi_volumes.1h37f1hz/home
/dev/mapper/vgsystem-srv ext4 496M 382M 79M 83% /tmp/kiwi_volumes.1h37f1hz/srv
/dev/mapper/vgsystem-var ext4 66M 30M 32M 49% /tmp/kiwi_volumes.1h37f1hz/var
tmpfs tmpfs 393M 0 393M 0% /run/user/0
config.xml (disk):
<type image="oem" filesystem="ext4" initrd_system="dracut" installiso="true" bootloader="grub2" kernelcmdline="splash lang=hu_HU nomodeset" mdraid="mirroring" firmware="efi">
<oemconfig>
<oem-systemsize>40960</oem-systemsize>
<oem-swap>true</oem-swap>
<oem-swapsize>2048</oem-swapsize>
<oem-device-filter>/dev/ram</oem-device-filter>
<oem-multipath-scan>false</oem-multipath-scan>
</oemconfig>
<!--
<machine memory="512" guestOS="suse" HWversion="4">
<vmdisk id="0" controller="ide"/>
<vmnic driver="e1000" interface="0" mode="bridged"/>
</machine>
-->
<systemdisk name="vgsystem" preferlvm="true">
<!-- <systemdisk name="vgsystem"> -->
<volume name="srv" freespace="6000M"/>
<volume name="var" freespace="6000M"/>
<volume name="home" freespace="6000M"/>
<volume name="@root" size="20000M"/>
<!-- <volume name="@root" freespace="10000M"/> -->
</systemdisk>
</type>
</preferences>
If it runs into a no space left issue it usually does not hang but just returns from rsync with an error. This is very strange. I'm still thinking it might be a conflict with lvm. /dev/mapper/system--disk2--build42-srv
is that also living in a logical volume ?
Building an lvm based image on a host that uses lvm for itself has caused problems in the past. But the reported hang would be a new thing. What does vgdisplay
and lvdisplay
show on your system ?
Yes /dev/mapper/system--disk2--build42-srv
is on an LV.
If I run the build process, the LV-s form the build are also showing up (if the build process isn't running, than only the system-disk2-build42
VG, and the /dev/system-disk2-build42/srv
LV are listed):
vgdisplay
--- Volume group ---
VG Name system-disk2-build42
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 9
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 110,00 GiB
PE Size 4,00 MiB
Total PE 28159
Alloc PE / Size 28159 / 110,00 GiB
Free PE / Size 0 / 0
VG UUID Y6WPdL-Grw8-aVdi-01mg-mpl1-6c31-7NB65q
--- Volume group ---
VG Name vgsystem
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 4
Max PV 0
Cur PV 1
Act PV 1
VG Size 6,87 GiB
PE Size 4,00 MiB
Total PE 1758
Alloc PE / Size 1746 / 6,82 GiB
Free PE / Size 12 / 48,00 MiB
VG UUID 5gW6ii-knj2-YrG5-XAkL-KbFW-LP8w-UDzV6I
lvdisplay
--- Logical volume ---
LV Path /dev/system-disk2-build42/srv
LV Name srv
VG Name system-disk2-build42
LV UUID aEVmte-toPu-Bu9a-CcG4-gdh2-PxXq-2lgs3L
LV Write Access read/write
LV Creation host, time build42, 2018-04-12 12:50:47 +0200
LV Status available
# open 1
LV Size 110,00 GiB
Current LE 28159
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:0
--- Logical volume ---
LV Path /dev/vgsystem/home
LV Name home
VG Name vgsystem
LV UUID hVMrhy-0zRj-O9CG-BzNL-Ekxo-vfMx-czAnk1
LV Write Access read/write
LV Creation host, time build15, 2019-04-23 11:09:39 +0200
LV Status available
# open 1
LV Size 108,00 MiB
Current LE 27
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:5
--- Logical volume ---
LV Path /dev/vgsystem/srv
LV Name srv
VG Name vgsystem
LV UUID ndNWMh-Tu0l-wNp4-fckK-34qS-FALC-1vhVFU
LV Write Access read/write
LV Creation host, time build15, 2019-04-23 11:09:39 +0200
LV Status available
# open 1
LV Size 520,00 MiB
Current LE 130
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:6
--- Logical volume ---
LV Path /dev/vgsystem/var
LV Name var
VG Name vgsystem
LV UUID mp2h3U-t7Kw-FCSg-pCdf-wOMi-p2Cz-BHlYrA
LV Write Access read/write
LV Creation host, time build15, 2019-04-23 11:09:40 +0200
LV Status available
# open 1
LV Size 72,00 MiB
Current LE 18
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:7
--- Logical volume ---
LV Path /dev/vgsystem/LVRoot
LV Name LVRoot
VG Name vgsystem
LV UUID hQjiYI-TPKm-55Tn-zbQY-Gu7x-pkLc-gMa7t1
LV Write Access read/write
LV Creation host, time build15, 2019-04-23 11:09:41 +0200
LV Status available
# open 1
LV Size 6,14 GiB
Current LE 1571
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:8
p.s. There was an earlier kiwi version, it had no problem with the same setup (maybe some of 9.14.x or 9.15.x).
I modified the build system, now it is a system withouth any LV, but the build process still hangs at the rsync phase.
If I run it manually:
rsync -a -H -X -A --one-file-system --exclude /image --exclude /.profile --exclude /.kconfig --exclude /.buildenv --exclude /var/cache/kiwi --exclude /boot/* --exclude /boot/.* --exclude /boot/efi/* --exclude /boot/efi/.* /srv/kiwi/huedu-4.0.4/build/image-root/ /tmp/kiwi_volumes.td4obmsq
I get the same as before:
rsync: mkstemp "/tmp/kiwi_volumes.td4obmsq/srv/www/roundcubemail/public_html/.index.php.v2zYTm" failed: No space left on device (28)
rsync: mkstemp "/tmp/kiwi_volumes.td4obmsq/srv/www/roundcubemail/vendor/.autoload.php.7fUvhA" failed: No space left on device (28)
rsync: mkstemp "/tmp/kiwi_volumes.td4obmsq/srv/www/squidanalyzer/.flotr2.js.dyj3EN" failed: No space left on device (28)
rsync: mkstemp "/tmp/kiwi_volumes.td4obmsq/srv/www/squidanalyzer/.sorttable.js.0u0D20" failed: No space left on device (28)
rsync: mkstemp "/tmp/kiwi_volumes.td4obmsq/srv/www/squidanalyzer/.squidanalyzer.css.wa9eqe" failed: No space left on device (28)
If you need the build sources for debuging, I can share them with you.
Thank for your help!
We can close this, it was some naughty FS bug. I tried earlier the xfs_repair -n
on the build system, but it didn't showed any errors. Today I deleted the build source directories, rsync-ed them again to the build system, and it doesn't hangs anymore.
Thanks much for hunting this down. I was spinning my head what in kiwi could have caused this as you mentioned it worked in a former version :)
Problem description
I try to build an oem image, and at a certain point the build process keeps hanging.
Expected behaviour
It should finish the build process without any problem.
Steps to reproduce the behaviour
Start a kiwi-ng build: kiwi-ng-3 --color-output --type oem system build --target-dir /srv/kiwi/this_build-4.0.4 --description . And if you let it run for 2 days long, it won't finish. The process should normally take less than 2 hours.
OS and Software information
additional info
There is no problem with free space, or RAM, the kiwi process hangs at rsync. Kiwi output: [ INFO ]: 10:44:08 | --> Syncing EFI boot data to EFI partition [ WARNING ]: 10:44:08 | Extended attributes not supported for target: /tmp/kiwi_mount_manager._l_wx67x [ INFO ]: 10:44:08 | --> Syncing boot data at extra partition [ INFO ]: 10:44:09 | --> Syncing root filesystem data
Kiwi log: INFO: 10:44:08 | --> GRUB_THEME:/boot/grub2/themes/openSUSE/theme.txt INFO: 10:44:08 | --> GRUB_TIMEOUT:10 INFO: 10:44:08 | --> GRUB_USE_INITRDEFI:true INFO: 10:44:08 | --> GRUB_USE_LINUXEFI:true INFO: 10:44:08 | Writing sysconfig bootloader file INFO: 10:44:08 | --> DEFAULT_APPEND:"splash lang=hu_HU nomodeset root=UUID=c543cb96-87fd-480e-9ab2-eda8f9901b9f rw rd.auto" INFO: 10:44:08 | --> FAILSAFE_APPEND:"splash lang=hu_HU nomodeset root=UUID=c543cb96-87fd-480e-9ab2-eda8f9901b9f rw rd.auto ide=nodma apm=off noresume edd=off nomodeset 3 rd.auto" INFO: 10:44:08 | --> LOADER_LOCATION:none INFO: 10:44:08 | --> LOADER_TYPE:grub2-efi INFO: 10:44:08 | Creating config.bootoptions INFO: 10:44:08 | Syncing system to image INFO: 10:44:08 | --> Syncing EFI boot data to EFI partition DEBUG: 10:44:08 | EXEC: [mountpoint -q /tmp/kiwi_mount_manager._l_wx67x] DEBUG: 10:44:08 | EXEC: [mount /dev/mapper/loop0p2 /tmp/kiwi_mount_manager._l_wx67x] WARNING: 10:44:08 | Extended attributes not supported for target: /tmp/kiwi_mount_manager._l_wx67x DEBUG: 10:44:08 | EXEC: [rsync -a -H --one-file-system /srv/kiwi/huedu-4.0.4/build/image-root/boot/efi/ /tmp/kiwi_mount_manager._l_wx67x] DEBUG: 10:44:08 | EXEC: [mountpoint -q /tmp/kiwi_mount_manager._l_wx67x] DEBUG: 10:44:08 | EXEC: [umount /tmp/kiwi_mount_manager._l_wx67x] INFO: 10:44:08 | --> Syncing boot data at extra partition DEBUG: 10:44:08 | EXEC: [mountpoint -q /tmp/kiwi_mountmanager.5mk1dyi] DEBUG: 10:44:08 | EXEC: [mount /dev/mapper/loop0p3 /tmp/kiwi_mountmanager.5mk1dyi] DEBUG: 10:44:08 | EXEC: [rsync -a -H -X -A --one-file-system --exclude /efi/ /srv/kiwi/huedu-4.0.4/build/image-root/boot/ /tmp/kiwi_mountmanager.5mk1dyi] DEBUG: 10:44:08 | EXEC: [mountpoint -q /tmp/kiwi_mountmanager.5mk1dyi] DEBUG: 10:44:08 | EXEC: [umount /tmp/kiwi_mountmanager.5mk1dyi] INFO: 10:44:09 | --> Syncing root filesystem data DEBUG: 10:44:09 | EXEC: [mountpoint -q /tmp/kiwi_volumes.h4rtd9e1] DEBUG: 10:44:09 | EXEC: [rsync -a -H -X -A --one-file-system --exclude /image --exclude /.profile --exclude /.kconfig --exclude /.buildenv --exclude /var/cache/kiwi --exclude /boot/ --exclude /boot/. --exclude /boot/efi/ --exclude /boot/efi/.* /srv/kiwi/huedu-4.0.4/build/image-root/ /tmp/kiwi_volumes.h4rtd9e1]