lx-brand: dataset's not mounted

[sjorge]

I added a delegated dataset to a lx branded zone. They do not seem to get mounted!

I can use /native/usr/sbin/zfs to create a few datasets

/native/usr/sbin/zfs create -o mountpoint=/srv/www zones/`/native/usr/bin/zonename`/data/www
/native/usr/sbin/zfs create -o mountpoint=/home -o canmount=no zones/`/native/usr/bin/zonename`/data/homes
/native/usr/sbin/zfs create -o quota=5G zones/`/native/usr/bin/zonename`/data/homes/sjorge

If you do this on a native zone, the stuff shows up under /srv/www, /home/sjorge,...

Manually running /native/usr/sbin/zfs mount -a seems mounts the stuff. But it is needed every reboot. /etc/mtab reflects this, however linux mount does not.

Some more integration for this would be awesome. Perhaps lxinit can take care of this?

[sjorge]

In addition to having lxinit mount all zfs datasets in the delegated dataset, it shoudl also be reflected in the kernel layer. Even if mounted with zfs mount -a, fstab (duh), mtab,... do not reflect it.

[drscream]

I more or less have the same issue with an LX branded zone (3da6330e-388d-11e6-b41b-d766707c6c3d). But for me it looks a little bit more strange during the creation of the zfs.

If I create a dataset and set the mountpoint it's mounted to that location:

$ UUID=$(mdata-get sdc:uuid)
$ DDS=zones/${UUID}/data
$ /native/sbin/zfs create ${DDS}/my-home
$ /native/sbin/zfs set mountpoint=/my-home ${DDS}/my-home

The folder is created and also everything looks okey:

$ ls -la /my-home/
total 17
drwxr-xr-x  2 root root  2 Sep  7 07:16 .
drwxr-xr-x 33 root root 33 Sep  7 07:16 ..

$ mount
...
zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home on /my-home type zfs (rw)
...

$ /native/sbin/zfs get mountpoint zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home
NAME                                                     PROPERTY    VALUE       SOURCE
zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home  mountpoint  /my-home    local

Now I create a test file in that folder and reboot the LX branded zone:

$ touch /my-home/test-file
$ reboot

After the reboot the file is gone and somehow also mount doesn't show the dataset to be mounted. I tried to mount it manually with zfs mount -a but I receive only an error.

$ ls -la /my-home/
total 17
drwxr-xr-x  2 root root  2 Sep  7 07:16 .
drwxr-xr-x 33 root root 33 Sep  7 07:16 ..

$ mount
/dev/zfsds0 on / type zfs (rw)
devtmpfs on /dev type devtmpfs (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /run type tmpfs (rw)
tmpfs on /run/lock type tmpfs (rw)
tmpfs on /sys/fs/cgroup type tmpfs (rw)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw)
zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data on /zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data type zfs (rw)
/native/usr on /native/usr type zfs (ro)

$ /native/usr/sbin/zfs mount -a
cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy

$ /native/sbin/zfs list
...
zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home   144K  97.0G   144K  /my-home

For me I'm not 100% sure if it's really mounted or not. Also I'm not sure what happen during the creation state if it somehow was mounted or not and if my file is gone.

[drscream]

I also tried to unmount the folder but it also failed with not currently mounted:

$ /native/sbin/zfs unmount zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home
cannot unmount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': not currently mounted

[jjelinek]

I am not sure what image you are using in your lx zone, but for the images we create at Joyent this is normally handled by the /etc/rc.local service that gets run by whatever service manager your image is using (upstart, systemd, etc). The rc.local script will run /lib/smartdc/mount-zfs which then runs this command "/native/usr/sbin/zfs mount -va". This behavior is essentially no different than for any native illumos boot, which will use SMF to run a service which runs that same command (zfs mount -va, although not using the /native path).

It sounds like your installed image is not performing this service on boot. If that is the case, you should create an appropriate script for your service manager to do this work and ensure that the service is enabled and runs during boot.

Jerry

On Wed, Sep 7, 2016 at 1:23 AM, Thomas Merkel notifications@github.com wrote:

I more or less have the same issue with an LX branded zone ( 3da6330e-388d-11e6-b41b-d766707c6c3d). But for me it looks a little bit more strange during the creation of the zfs.

If I create a dataset and set the mountpoint it's mounted to that location:

$ UUID=$(mdata-get sdc:uuid) $ DDS=zones/${UUID}/data $ /native/sbin/zfs create ${DDS}/my-home $ /native/sbin/zfs set mountpoint=/my-home ${DDS}/my-home

The folder is created and also everything looks okey:

$ ls -la /my-home/ total 17 drwxr-xr-x 2 root root 2 Sep 7 07:16 . drwxr-xr-x 33 root root 33 Sep 7 07:16 ..

$ mount ... zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home on /my-home type zfs (rw) ...

$ /native/sbin/zfs get mountpoint zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home NAME PROPERTY VALUE SOURCE zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home mountpoint /my-home local

Now I create a test file in that folder and reboot the LX branded zone:

$ touch /my-home/test-file $ reboot

After the reboot the file is gone and somehow also mount doesn't show the dataset to be mounted. I tried to mount it manually with zfs mount -a but I receive only an error.

$ ls -la /my-home/ total 17 drwxr-xr-x 2 root root 2 Sep 7 07:16 . drwxr-xr-x 33 root root 33 Sep 7 07:16 ..

$ mount /dev/zfsds0 on / type zfs (rw) devtmpfs on /dev type devtmpfs (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) tmpfs on /dev/shm type tmpfs (rw) tmpfs on /run type tmpfs (rw) tmpfs on /run/lock type tmpfs (rw) tmpfs on /sys/fs/cgroup type tmpfs (rw) cgroup on /sys/fs/cgroup/systemd type cgroup (rw) zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data on /zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data type zfs (rw) /native/usr on /native/usr type zfs (ro)

$ /native/usr/sbin/zfs mount -a cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy

$ /native/sbin/zfs list ... zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home 144K 97.0G 144K /my-home

For me I'm not 100% sure if it's really mounted or not. Also I'm not sure what happen during the creation state if it somehow was mounted or not and if my file is gone.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/joyent/illumos-joyent/issues/55#issuecomment-245197227, or mute the thread https://github.com/notifications/unsubscribe-auth/AApIw6YJlhovT9WDSxmaLI6FBzeRGMtcks5qnmaNgaJpZM4EtFSl .

[drscream]

Thanks for your feedback, I using the official Debian LX-Branded Zone from Joyent 3da6330e-388d-11e6-b41b-d766707c6c3d. I also have the scripts installed and I also assume they are working.

Anyway the problem is during creating the dataset / extra volume. At this time it looks mounted and I store data there, after the reboot the data is gone. I think it's still there but somehow the zfs mount -va command mounted it over the existing directory.

So after the zfs create and zfs set mountpoint command the FS is not mounted but shown as mounted via the mount command.

I've also tried to disable the /etc/rc.local part which do the zfs mount -va and run it manually after boot which result also in the following error:

cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy

Then I've tried to do some modifications for the mountpoint so what I did:

$ /native/sbin/zfs set mountpoint=none zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home
$ reboot
$ /native/sbin/zfs set mountpoint=/my-home zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home

cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy
property may be set but unable to remount filesystem

[drscream]

I think (I hope I think) I isolated the problem. /lib/smartdc/mount-zfs (or basically the mount command) fails if the folder /my-home already exists. Well a zfs mount create the folder to mount it successfully - but after the reboot the folder exists so the mount command fail.

$ /lib/smartdc/mount-zfs
(info) - /lib/smartdc/mount-zfs - Mounting all ZFS filesystems
cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy
(info) - /lib/smartdc/mount-zfs - WARNING: /native/usr/sbin/zfs mount -va failed: exited     1

I've removed the /my-home folder and executed the /lib/smartdc/mount-zfs again.

$ rm -rf /my-home
$ /lib/smartdc/mount-zfs

And my test-file appear again and also the df -h & mount output is working.

$ ls -la /my-home/
total 18
drwxr-xr-x  3 root root  3 Sep  7 07:20 .
drwxr-xr-x 33 root root 33 Sep  7 14:12 ..
-rw-r--r--  1 root root  0 Sep  7 07:20 test-file
dr-xr-xr-x  4 root root  4 Sep  7 07:16 .zfs

[jjelinek]

Instead of removing the entire mount point, can you try one other test. Can you leave the directory in place, but see if everything works when the directory is empty vs. if things fail if there is a file/subdirectory inside the mount point directory.

Thanks, Jerry

On Wed, Sep 7, 2016 at 8:13 AM, Thomas Merkel notifications@github.com wrote:

I think (I hope I think) I isolated the problem. /lib/smartdc/mount-zfs (or basically the mount command) fails if the folder /my-home already exists. Well a zfs mount create the folder to mount it successfully - but after the reboot the folder exists so the mount command fail.

$ /lib/smartdc/mount-zfs (info) - /lib/smartdc/mount-zfs - Mounting all ZFS filesystems cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy (info) - /lib/smartdc/mount-zfs - WARNING: /native/usr/sbin/zfs mount -va failed: exited 1

I've removed the /my-home folder and executed the /lib/smartdc/mount-zfs again.

$ rm -rf /my-home $ /lib/smartdc/mount-zfs

And my test-file appear again and also the df -h & mount output is working.

$ ls -la /my-home/ total 18 drwxr-xr-x 3 root root 3 Sep 7 07:20 . drwxr-xr-x 33 root root 33 Sep 7 14:12 .. -rw-r--r-- 1 root root 0 Sep 7 07:20 test-file dr-xr-xr-x 4 root root 4 Sep 7 07:16 .zfs

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/joyent/illumos-joyent/issues/55#issuecomment-245293439, or mute the thread https://github.com/notifications/unsubscribe-auth/AApIw9nRc7Q9g72r0mCzPdwS560FaR48ks5qnsaPgaJpZM4EtFSl .

[drscream]

I tried that already, and it failed both times:

$ ls -la /my-home/
total 17
drwxr-xr-x  2 root root  2 Sep  7 14:12 .
drwxr-xr-x 33 root root 33 Sep  7 14:12 ..

$ /lib/smartdc/mount-zfs
cannot mount 'zones/b5662993-1f86-c0ae-845a-c5cfbecf79a6/data/my-home': mountpoint or dataset is busy

If I touch a file in the folder (so it's not empty) I receive the following error:

$ /lib/smartdc/mount-zfs
cannot mount '/my-home': directory is not empty

[blackwood821]

I am experiencing this issue as well. Any update on this?

[jasonbking]

This is likely OS-8054.