Docker starts before the ZFS mount happend

[Skaronator]

Looks like the Docker Plugin/System was starting faster than the ZFS mount after a reboot.

My ZFS Raid is mounted on /StorageZFS and my Docker folder is on /StorageZFS/Docker. After a System Reboot Docker was started before the ZFS Raid was mounted so it deployed the files on /StorageZFS/Docker but at this point it was the System SSD and not the ZFS Raid. So and now ZFS tried to mount on /StorageZFS but failed because the folder was not empty.

It took me over 2 hours to get my Raid working again. I tried to uninstall the Plugin but it gave me a gateway error. [... Long story short]: I attached another drive and moved the docker folder to the new drive and then I was able to uninstall docker.

[subzero79]

Post in hastebin or gist

journalctl -b

What version of Omv is this ?

[Skaronator]

Its OMV 3, Debian 8.

I rebooted the machine couple times to fix the issues so the journalctl log doesn't contain the infos, Are there older longs available? Google says no?

[subzero79]

Doesn't matter, I want to see the sequence of services started. Docker starts after network if I am not mistaken, I would tend to thing all zfs mount services start very early, before network. I know this was a very old issue in wheezy (solved) not now.

[Skaronator]

Alright so I'm going to install Docker again. It doesn't matter if I use the ZFS Raid or just a other drive, right?

[subzero79]

You can use the default location /var/lib/docker, For now don't use the zfs raid

I think I know the problem, did you use the storage selector with shared folder in the plugin ?

[Skaronator]

Yes I created a shared folder and used them for Docker.

Here is the full log: https://pastebin.com/v7iAFMYE

LINE 1062; Jul 08 23:31:50 OMV-NAS systemd[1]: Started Import ZFS pools by device scanning. LINE 1265: Jul 08 23:31:53 OMV-NAS systemd[1]: Starting Docker Application Container Engine... LINE 1295: Jul 08 23:31:53 OMV-NAS systemd[1]: Starting ZFS file system shares... LINE 1304: Jul 08 23:31:53 OMV-NAS systemd[1]: Reached target ZFS startup target.

Looks like ZFS is started before it starts Docker but it takes longer to import the raid.

[subzero79]

I think the problem is that shared folder selection it creates a mount bind to /var/lib/docker since is inside zfs, systemd fstab cannot create the dependency. Imo that folder selection should be removed. If you want to keep using docker, don't use the shared folder, create a symlink from /var/lib/docker pointing to /StorageZfs/Docker

[Skaronator]

Alrighty. Just one question, how is Docker started via Systemd?

[subzero79]

Alrighty. Just one question, how is Docker started via Systemd?

Yes, the plugin adds a drop-in conf to enable the tcp api on localhost

[subzero79]

BTW zfs file system shares are not the mounted pool, you can clearly see in the logs that zfs is mounted before docker. Test the symlink and report back

[Skaronator]

I'll test this tomorrow, currently busy.

If I can't make it in time then on the next weekend. Don't want to touch a remote system.

[Skaronator]

Looks like symlinks works fine. Would it be possible to change the plugin logic to create a symlink when you select a shared folder? I think the Plex Plugin did this as well for a while until Plex added support for custom path but not sure.

service docker stop
rm -rf /var/lib/docker
ln -s /StorageZFS/Docker /var/lib/docker
service docker start

[subzero79]

It is possible. But for now we need to think how to handle the upgrades.

[Skaronator]

There is an entire doc page for Docker on ZFS: https://docs.docker.com/engine/userguide/storagedriver/zfs-driver/#configure-docker-with-the-zfs-storage-driver

Best would be to mount the ZFS FS on the docker location. I'm now on Debian 9 without OpenMediaVault so thats why I saw that.

[subzero79]

Closed with 471ed274a3dc82e729ff645565b2783fa41f72e8

the plugin will no longer use a bind mount. Will pass the -g parameter directly to dockerd