search

kamilion / customizer

Ubuntu Live CD remastering tool
GNU General Public License v2.0
306 stars 94 forks source link

Exponential mount points when rebuilding ISO #207

Open pdjstone opened 5 years ago

pdjstone commented 5 years ago

There seems to be a problem whereby Customizer is causing mounts to be created recursively, leading to my system becoming unstable. If I run customizer -r once (it successfully builds the ISO) then run mount I get the following mounts related to Customizer:

/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
proc on /home/customizer/FileSystem/proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /home/customizer/FileSystem/dev type devtmpfs (rw,nosuid,relatime,size=8124556k,nr_inodes=2031139,mode=755)
devpts on /home/customizer/FileSystem/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /home/customizer/FileSystem/dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
sysfs on /home/customizer/FileSystem/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
tmpfs on /home/customizer/FileSystem/run/dbus type tmpfs (rw,nosuid,noexec,relatime,size=1631356k,mode=755)

If I run customizer -r a second time, I get the following mounts:

/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
proc on /home/customizer/FileSystem/proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /home/customizer/FileSystem/dev type devtmpfs (rw,nosuid,relatime,size=8124556k,nr_inodes=2031139,mode=755)
devpts on /home/customizer/FileSystem/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /home/customizer/FileSystem/dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
sysfs on /home/customizer/FileSystem/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /tmp type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /home/customizer/FileSystem/var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/ubuntu--vg-root on /var/lib/dbus type ext4 (rw,relatime,errors=remount-ro,data=ordered)
tmpfs on /home/customizer/FileSystem/run/dbus type tmpfs (rw,nosuid,noexec,relatime,size=1631356k,mode=755)

Each time I rebuild the ISO I get exponentially more mounts. This causes a bunch of Gnome system services to start using 100% (presumably they're enumerating mount points) at which point my system crawls to a halt.

I'm running Ubuntu 18.04 with LVM/LUKS (the default disk-encryption install option).

pdjstone commented 5 years ago

It seems that the number of mounts created by customizer multiplies by 8 each time I rebuild the ISO.

After rebuilding the ISO twice (as above) I ran mount | grep customizer | wc -l and got 256. While rebuilding a third time, I counted the mounts as it built and got 512...1030...2048...4096.

pdjstone commented 5 years ago

I enabled MOUNT_DEBUG in src/lib/misc.py and got the following output;

  => Gathering information
  => Mounting --bind /proc: /home/customizer/FileSystem/proc
  => Mounting --bind /dev: /home/customizer/FileSystem/dev
  => Mounting --bind /dev/pts: /home/customizer/FileSystem/dev/pts
  => Mounting --bind /dev/shm: /home/customizer/FileSystem/dev/shm
  => Mounting --bind /sys: /home/customizer/FileSystem/sys
  => Mounting --bind /tmp: /home/customizer/FileSystem/tmp
  => Mounting --bind /var/lib/dbus: /home/customizer/FileSystem/var/lib/dbus
  => Mounting --bind /run/dbus: /home/customizer/FileSystem/run/dbus
  => Unmounting -f -l: /home/customizer/FileSystem/run/dbus
  => Unmounting -f -l: /home/customizer/FileSystem/sys
  => Unmounting -f -l: /home/customizer/FileSystem/dev/shm
  => Unmounting -f -l: /home/customizer/FileSystem/dev/pts
  => Unmounting -f -l: /home/customizer/FileSystem/dev
  => Unmounting -f -l: /home/customizer/FileSystem/proc

The issue seems to be that os.path.ismount returns False for /home/customizer/FileSystem/tmp and /home/customizer/FileSystem/var/lib/dbus even though they clearly are mounted. The python docs for ismount say that "it is not able to reliably detect bind mounts on the same filesystem" - so I guess that's the problem.

If I remove the ismount check everything seems to work correctly.

kamilion commented 5 years ago

Wow, thanks for debugging that for me, it's happened in the past and I had no idea where to start.

I'll see what I can think of to rework how customizer keeps track of mounts so ismount isn't relied on.