search

vmware / open-vm-tools

Official repository of VMware open-vm-tools project
http://sourceforge.net/projects/open-vm-tools/
2.24k stars 426 forks source link

deadlock caused by loop device #6

Open rjokl opened 9 years ago

rjokl commented 9 years ago

Have mounted loop device: mount loop.img loopmnt mount ... /dev/sdb1 on /home/rjokl/disk type ext3 (rw,relatime,data=ordered) /home/rjokl/disk/loop.img on /home/rjokl/loopmnt type ext3 (rw,relatime,data=ordered)

Now when you try to quiesce disks, vmtoolsd list all mounted FSs and call ioctl(fd, FIFREEZE) on them, one by one. However, that ioctl will hang on the loop device if the parent device is already frozen. It can be resumed by calling ioctl(fd, FITHAW) from another process, but that is not the case of vmtoolsd.

Result: guest is completely deadlocked.

I see several options to fix the issue:

  1. if the ioctl is not supposed to lock, then it is bug in kernel. I don't know the restrictions of ioctl usage.
  2. exclude loop devices from quiesce operation. underlaying block device is frozen, so the FS is probably consistent, but I'm not sure if this assumption correct.
  3. freeze loop device first and thaw them in reverse order (otherwise thaw ioctl will lock too)
dyno commented 9 years ago

file an internal bug and should be fixed in next push. most likely solution 2 will be picked. thanks for the report.

ravindravmw commented 8 years ago

We have fixed it internally. The fix will be available with next update of open-vm-tools.

Freeze commented 7 years ago

Do we have an ETA for that? This bug is proving challenging for our RHEL7 rollout.

oliverkurth commented 7 years ago

We have already released VMware Tools 10.1.0. The new version of open-vm-tools will be available very soon (a few days).

Freeze commented 7 years ago

I've just tested with version 10.0.5 available in the RHEL7.3 update. Will 10.1.0 be coming to the Red Hat repositories shortly as well, or are we going to need to compile this ourselves?

ravindravmw commented 7 years ago

@hooolden Red Hat usually picks newer version with their new release. For quicker resolution, could you please raise it with Red Hat? They could either build 10.1.0 or patch the 10.0.5 version they have.

nerionsis commented 6 years ago

Hello,

I have several cPanel servers (with and without CloudLinux) with CentOS 7 and the last open-vm-tools version available from CentOS repository (open-vm-tools-10.1.10-3.el7_5.1.x86_64), and it seems that the issue with loop device is not solved. In fact, the problem happens when we have configured /tmp as a partition (mapped to a file).

This is the answer we have received from CloudLinux developers and the temporary workaround:

"This confirms that the issue is not related to CloudLinux, like our kernel developers suggested. The problem is that before freezing virtual disk, all file systems have to be frozen first. And it appears that VMware does not freeze /tmp file system while making a snapshot. This causes a deadlock if something tries to write to /tmp while snapshot is being created, because /tmp is mapped to a file on root file system and it is being frozen, so ext4 driver waits for it to unfreeze to flush data. There is nothing we can do about it. I suggest that you report this issue to VMware. Alternatively you can disable securetmp script provided by cPanel that creates this /tmp mount. To disable it just run touch /var/cpanel/version/securetmp_disabled and then remove /tmp entry from /etc/fstab and reboot server. Documentation: https://documentation.cpanel.net/display/CKB/Tips+to+Make+Your+Server+More+Secure"

We have tested uninstalling the official open-vm-tools released by CentOS and installing the last open-vm-tools from your repository (10.3.0), and the freeze issue keeps happening.

ravindravmw commented 6 years ago

@nerionsis could you please run following bash script in your VM as root user and collect the output?

losetup -al cat > /tmp/mountlist.c << EOF

include

include

static void printMount(struct mntent * mnt) { printf("fsname=%s, dir=%s, type=%s, opts=%s, freq=%d, passno=%d\n", mnt->mnt_fsname, mnt->mnt_dir, mnt->mnt_type, mnt->mnt_opts, mnt->mnt_freq, mnt->mnt_passno); }

int main() { FILE mounts; struct mntent mntinfo;

mounts = setmntent(MOUNTED, "r"); while ((mntinfo = getmntent(mounts)) != NULL) { printMount(mntinfo); } endmntent(mounts); } EOF gcc -o /tmp/mountlist /tmp/mountlist.c /tmp/mountlist rm /tmp/mountlist.c /tmp/mountlist

nerionsis commented 6 years ago

This is the output from a CloudLinux + cPanel server (CentOS 7.5 - Kernel 3.10.0-714.10.2.lve1.5.17.el7.x86_64 and open-vm-tools-10.1.10-3.el7_5.1.x86_64)

NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE /dev/loop0 0 0 1 0 /usr/tmpDSK fsname=rootfs, dir=/, type=rootfs, opts=rw, freq=0, passno=0 fsname=sysfs, dir=/sys, type=sysfs, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=proc, dir=/proc, type=proc, opts=rw,nosuid,nodev,noexec,relatime,gid=1001,hidepid=2, freq=0, passno=0 fsname=devtmpfs, dir=/dev, type=devtmpfs, opts=rw,nosuid,size=1923820k,nr_inodes=480955,mode=755, freq=0, passno=0 fsname=securityfs, dir=/sys/kernel/security, type=securityfs, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=tmpfs, dir=/dev/shm, type=tmpfs, opts=rw,nosuid,nodev, freq=0, passno=0 fsname=devpts, dir=/dev/pts, type=devpts, opts=rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000, freq=0, passno=0 fsname=tmpfs, dir=/run, type=tmpfs, opts=rw,nosuid,nodev,mode=755, freq=0, passno=0 fsname=tmpfs, dir=/sys/fs/cgroup, type=tmpfs, opts=rw,nosuid,nodev,noexec,relatime,mode=755, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/systemd, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd, freq=0, passno=0 fsname=pstore, dir=/sys/fs/pstore, type=pstore, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=configfs, dir=/sys/kernel/config, type=configfs, opts=rw,relatime, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/, type=xfs, opts=rw,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=systemd-1, dir=/proc/sys/fs/binfmt_misc, type=autofs, opts=rw,relatime,fd=24,pipe_ino=12545,pgrp=1,timeout=0,minproto=5,maxproto=5,direct, freq=0, passno=0 fsname=mqueue, dir=/dev/mqueue, type=mqueue, opts=rw,relatime, freq=0, passno=0 fsname=debugfs, dir=/sys/kernel/debug, type=debugfs, opts=rw,relatime, freq=0, passno=0 fsname=hugetlbfs, dir=/dev/hugepages, type=hugetlbfs, opts=rw,relatime, freq=0, passno=0 fsname=/dev/sda1, dir=/boot, type=xfs, opts=rw,relatime,attr2,inode64,noquota, freq=0, passno=0 fsname=sunrpc, dir=/var/lib/nfs/rpc_pipefs, type=rpc_pipefs, opts=rw,relatime, freq=0, passno=0 fsname=/dev/loop0, dir=/tmp, type=ext3, opts=rw,nosuid,noexec,relatime,data=ordered, freq=0, passno=0 fsname=/dev/loop0, dir=/var/tmp, type=ext3, opts=rw,nosuid,noexec,relatime,data=ordered, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/freezer, type=cgroup, opts=rw,relatime,freezer, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/devices, type=cgroup, opts=rw,relatime,devices, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/cpu,cpuacct, type=cgroup, opts=rw,relatime,cpuacct,cpu, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/cpuset, type=cgroup, opts=rw,relatime,cpuset, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/memory, type=cgroup, opts=rw,relatime,memory, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=devpts, dir=/usr/share/cagefs-skeleton/dev/pts, type=devpts, opts=rw,nosuid,relatime,gid=5,mode=620,ptmxmode=000, freq=0, passno=0 fsname=tmpfs, dir=/usr/share/cagefs-skeleton/dev/shm, type=tmpfs, opts=rw,nosuid,relatime, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/lib, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/lib64, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/include, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/lib, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/lib64, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/apache/domlogs, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/bin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/lib, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/perl, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/php, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/3rdparty/share, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/Cpanel, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/sys_cpanel/boxtrapper-message, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/local/cpanel/var, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/share/l.v.e-manager/cl.nodejs, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/share/locale, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/share/terminfo, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/usr/share/zoneinfo, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/lib/mysql, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/lib/proxyexec/cagefs.sock, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/lib/spamassassin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=tmpfs, dir=/usr/share/cagefs-skeleton/var/run/dbus, type=tmpfs, opts=rw,nosuid,relatime,mode=755, freq=0, passno=0 fsname=tmpfs, dir=/usr/share/cagefs-skeleton/var/run/nscd, type=tmpfs, opts=rw,nosuid,relatime,mode=755, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/spool/at, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/www/cgi-bin, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/www/html, type=xfs, opts=rw,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php70/root/usr/bin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php70/root/etc, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php56/root/usr/bin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php56/root/etc, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php54/root/usr/bin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php54/root/etc, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php55/root/usr/bin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php55/root/etc, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php71/root/usr/bin, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/opt/cpanel/ea-php71/root/etc, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/centos_eXXXX-root, dir=/usr/share/cagefs-skeleton/var/lve/lveinfo.ver.cagefs, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=proc, dir=/usr/share/cagefs-skeleton/proc, type=proc, opts=rw,nosuid,relatime,gid=1001,hidepid=2, freq=0, passno=0 fsname=systemd-1, dir=/usr/share/cagefs-skeleton/proc/sys/fs/binfmt_misc, type=autofs, opts=rw,nosuid,relatime,fd=24,pipe_ino=12545,pgrp=1,timeout=0,minproto=5,maxproto=5,direct, freq=0, passno=0 fsname=nfsserver.domain.com:/backup2/eXXXX, dir=/backupnfs, type=nfs4, opts=rw,sync,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.2,local_lock=none,addr=A.B.C.D, freq=0, passno=0 fsname=tmpfs, dir=/run/user/0, type=tmpfs, opts=rw,nosuid,nodev,relatime,size=387036k,mode=700, freq=0, passno=0

And this is the output from cPanel server (CentOS 7.5 - Kernel 3.10.0-862.6.3.el7.x86_64 and open-vm-tools-10.1.10-3.el7_5.1.x86_64):

NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE /dev/loop0 0 0 1 0 /usr/tmpDSK fsname=rootfs, dir=/, type=rootfs, opts=rw, freq=0, passno=0 fsname=sysfs, dir=/sys, type=sysfs, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=proc, dir=/proc, type=proc, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=devtmpfs, dir=/dev, type=devtmpfs, opts=rw,nosuid,size=1924244k,nr_inodes=481061,mode=755, freq=0, passno=0 fsname=securityfs, dir=/sys/kernel/security, type=securityfs, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=tmpfs, dir=/dev/shm, type=tmpfs, opts=rw,nosuid,nodev, freq=0, passno=0 fsname=devpts, dir=/dev/pts, type=devpts, opts=rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000, freq=0, passno=0 fsname=tmpfs, dir=/run, type=tmpfs, opts=rw,nosuid,nodev,mode=755, freq=0, passno=0 fsname=tmpfs, dir=/sys/fs/cgroup, type=tmpfs, opts=ro,nosuid,nodev,noexec,mode=755, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/systemd, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd, freq=0, passno=0 fsname=pstore, dir=/sys/fs/pstore, type=pstore, opts=rw,nosuid,nodev,noexec,relatime, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/net_cls,net_prio, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,net_prio,net_cls, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/cpuset, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,cpuset, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/pids, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,pids, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/cpu,cpuacct, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,cpuacct,cpu, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/freezer, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,freezer, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/hugetlb, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,hugetlb, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/devices, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,devices, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/blkio, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,blkio, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/memory, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,memory, freq=0, passno=0 fsname=cgroup, dir=/sys/fs/cgroup/perf_event, type=cgroup, opts=rw,nosuid,nodev,noexec,relatime,perf_event, freq=0, passno=0 fsname=configfs, dir=/sys/kernel/config, type=configfs, opts=rw,relatime, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/, type=xfs, opts=rw,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=systemd-1, dir=/proc/sys/fs/binfmt_misc, type=autofs, opts=rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11136, freq=0, passno=0 fsname=mqueue, dir=/dev/mqueue, type=mqueue, opts=rw,relatime, freq=0, passno=0 fsname=hugetlbfs, dir=/dev/hugepages, type=hugetlbfs, opts=rw,relatime, freq=0, passno=0 fsname=debugfs, dir=/sys/kernel/debug, type=debugfs, opts=rw,relatime, freq=0, passno=0 fsname=/dev/sda1, dir=/boot, type=xfs, opts=rw,relatime,attr2,inode64,noquota, freq=0, passno=0 fsname=/dev/loop0, dir=/tmp, type=ext3, opts=rw,nosuid,noexec,relatime,data=ordered, freq=0, passno=0 fsname=/dev/loop0, dir=/var/tmp, type=ext3, opts=rw,nosuid,noexec,relatime,data=ordered, freq=0, passno=0 fsname=devtmpfs, dir=/home/virtfs/drsagrer/dev, type=devtmpfs, opts=rw,nosuid,noexec,size=1924244k,nr_inodes=481061,mode=755, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/opt, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/loop0, dir=/home/virtfs/drsagrer/tmp, type=ext3, opts=rw,nosuid,noexec,relatime,data=ordered, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/usr, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/var, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/etc/scl, type=xfs, opts=ro,nosuid,noexec,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/var/log, type=xfs, opts=rw,nosuid,noexec,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/loop0, dir=/home/virtfs/drsagrer/var/tmp, type=ext3, opts=rw,nosuid,noexec,relatime,data=ordered, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/etc/mail, type=xfs, opts=rw,nosuid,noexec,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/usr/sbin, type=xfs, opts=ro,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/var/spool, type=xfs, opts=rw,nosuid,noexec,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/etc/apache2, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/etc/pki/tls, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/etc/alternatives, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/etc/pki/ca-trust, type=xfs, opts=ro,nosuid,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/var/cpanel/php/sessions, type=xfs, opts=rw,nosuid,noexec,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/var/cpanel/email_send_limits, type=xfs, opts=rw,nosuid,noexec,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=/dev/mapper/vg_00-root, dir=/home/virtfs/drsagrer/home/drsagrer, type=xfs, opts=rw,relatime,attr2,inode64,usrquota, freq=0, passno=0 fsname=tmpfs, dir=/run/user/0, type=tmpfs, opts=rw,nosuid,nodev,relatime,size=387260k,mode=700, freq=0, passno=0