if debootstick fails it may sometimes leave residues like mounted filesystems

[unki]

Something I've observed when debootstick gets - whyever - interrupted during a build. In my case it was probably the /tmp filesystem that was running full (once again... now I've set TMPDIR to somewhere else) and some tasks like apt-get were still performing in the background.

debootstick already does a great job trapping these interruptions and invokes its on_exit() function, starting to undo all the changes that it has previously made.

It my case, it wasn't successful here - perhaps failed to umount because the kernel was unable to write dirty pages back into the loopback-mounted image-file as there was not more free space in /tmp available for this action.

And in this situation some residues have been left - next to the tmp-directory that debootstick used for building also all the mount points like /proc and /sys that it has mounted in there.

It was looking like this:

/dev/mapper/DRAFT--8ffb870adce14fb3ab35847eaa3c0ac2-ROOT on /tmp/tmp.dbstck.crZjW.d/draft/lvroot type ext4 (rw,relatime,data=ordered)
/dev/mapper/loop0p1 on /tmp/tmp.dbstck.crZjW.d/draft/efipart type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=utf8,shortname=mixed,errors=remount-ro)
/dev/mapper/vg_system-root on /tmp/tmp.dbstck.crZjW.d/draft/lvroot/etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
none on /tmp/tmp.dbstck.crZjW.d/draft/lvroot/proc type proc (rw,relatime)
none on /tmp/tmp.dbstck.crZjW.d/draft/lvroot/dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=4125030,mode=755)
none on /tmp/tmp.dbstck.crZjW.d/draft/lvroot/dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
none on /tmp/tmp.dbstck.crZjW.d/draft/lvroot/sys type sysfs (rw,relatime)

If now someone overlooks that there is still something mounted in $TMPDIR and just invokes rm -rf on this directory, he may end up with a painfully damaged host-system.

rm -rf traverses down all files and directories and when hitting /proc, /sys, etc. it will also start to delete files, directories, device-nodes, etc. of the building-hosts that these filesystems are referring to (like all stuff in /proc/*/fd/).

While you could argue that this is the users responsibility to carefully cleanup, perhaps it would be good too, to display a big, fat warning and show some commands that need to be manually invoked to cleanup, if debootstick was unable to do so. At least in my case the minimum set was:

user@host:~/usbstick$ sudo lsof | grep /tmp/tmp.dbstck.crZjW.d
... 
user@host:~/usbstick$ sudo kill $(process that were stuck in device state)

user@host:~/usbstick$ sudo umount /tmp/tmp.dbstck.crZjW.d/draft/lvroot/sys /tmp/tmp.dbstck.crZjW.d/draft/lvroot/dev/pts  /tmp/tmp.dbstck.crZjW.d/draft/lvroot/dev /tmp/tmp.dbstck.crZjW.d/draft/lvroot/proc /tmp/tmp.dbstck.crZjW.d/draft/lvroot/etc/resolv.conf /tmp/tmp.dbstck.crZjW.d/draft/efipart /tmp/tmp.dbstck.crZjW.d/draft/lvroot

user@host:~/usbstick$ sudo dmsetup remove /dev/mapper/DRAFT--8ffb870adce14fb3ab35847eaa3c0ac2-ROOT
user@host:~/usbstick$ sudo dmsetup remove /dev/mapper/loop0p1 
user@host:~/usbstick$ sudo dmsetup remove /dev/mapper/loop0p2
user@host:~/usbstick$ sudo dmsetup remove /dev/mapper/loop0p3

user@host:~/usbstick$ sudo losetup 
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /tmp/tmp.dbstck.crZjW.d/draft/file
user@host:~/usbstick$ sudo losetup -D
user@host:~/usbstick$ sudo losetup 
user@host:~/usbstick$ 

To summarize: if an error occurs in on_exit() or any of its called sub functions, debootstick could display what and where residues are left - and shows what could be done to cleanup manually.

IMHO you will never be able to cover all the possible fault situations that could occur - deboostick is already great in doing what it can to cleanup as much as possible. But if this fails, it could advice the user, how to follow up here.

[eduble]

That's interesting. It has been quite long since I got this kind of issues myself. My point of view is the following:

• if we know how to cleanup properly, instead of proposing it to the user, why not modify debootstick's cleanup procedures to make them more robust?
• if these robust procedures still fail, I agree we should detect it and warn the user. I will check how we could handle such an issue in the cleanup phase.

I tried to reproduce the issue, mounting the temporary directory using tmpfs with a low size limit. Errors occured, of course, but debootstick could still cleanup properly. If you have more details about the environment that caused issues, It would help if you could describe it. Thanks !

[eduble]

Hi, commit 8774a20f5782d07e0930feb38567d41d684bba83 should have fixed last corner cases, I am closing the issue.