Add support /dev/fuse

[cedvan]

Hi,

Please possible add support fuse module ?

Thank you

[ailispaw]

Hi @cedvan , Thank you for opening an issue.

How can I add that module and test it? Could you make a PR for it?

[cedvan]

I don't know, but I can try

[ailispaw]

If all you need is fuse in user land, you can install with https://github.com/ailispaw/docker-root-pkg.

$ sudo pkg install libfuse

It will install /usr/bin/fusermount and /usr/lib/libfuse.so*.

If you need any kernel configuration for it, please let me know.

[ailispaw]

May need CONFIG_FUSE_FS in kernel.config, too.

[ailispaw]

https://github.com/libfuse/libfuse

[cedvan]

Thank you !

I had install fuse with sudo pkg install libfuse.

Result :

$ modprobe -l |grep fuse
kernel/drivers/message/fusion/mptbase.ko
kernel/drivers/message/fusion/mptscsih.ko
kernel/drivers/message/fusion/mptspi.ko
kernel/drivers/message/fusion/mptsas.ko
kernel/drivers/message/fusion/mptctl.ko
$ cat /dev/fuse
cat: can't open '/dev/fuse': No such file or directory

Need CONFIG_FUSE_FS to enable fuse ? Where is kernel.config file please ?

[ailispaw]

https://github.com/ailispaw/docker-root/blob/master/configs/kernel.config#L2028

[ailispaw]

sudo pkg install libfuse installs only modules in userspace. You need CONFIG_FUSE_FS for kernel modules and devices.

[ailispaw]

diff --git a/configs/buildroot.config b/configs/buildroot.config
index 2a62e5c..571277b 100644
--- a/configs/buildroot.config
+++ b/configs/buildroot.config
@@ -1065,7 +1065,7 @@ BR2_PACKAGE_OPENSSL=y
 # BR2_PACKAGE_GAMIN is not set
 # BR2_PACKAGE_LIBCONFIG is not set
 # BR2_PACKAGE_LIBCONFUSE is not set
-# BR2_PACKAGE_LIBFUSE is not set
+BR2_PACKAGE_LIBFUSE=y
 # BR2_PACKAGE_LIBLOCKFILE is not set
 # BR2_PACKAGE_LIBNFS is not set
 # BR2_PACKAGE_LIBSYSFS is not set
diff --git a/configs/kernel.config b/configs/kernel.config
index 2fad792..602e5dc 100644
--- a/configs/kernel.config
+++ b/configs/kernel.config
@@ -2025,7 +2025,8 @@ CONFIG_INOTIFY_USER=y
 # CONFIG_QUOTA is not set
 # CONFIG_QUOTACTL is not set
 # CONFIG_AUTOFS4_FS is not set
-# CONFIG_FUSE_FS is not set
+CONFIG_FUSE_FS=y
+# CONFIG_CUSE is not set
 CONFIG_OVERLAY_FS=y

 #

[cedvan]

Ok thank you very much !

I will create a PR as soon as possible ;)

[ailispaw]

If that modification doesn't make kernel bigger significantly, I will add CONFIG_FUSE_FS in the next release.

[ailispaw]

BTW, what fuse will benefit and what is your use case?

[cedvan]

I need fuse to support glusterfs :)

[ailispaw]

Is glusterfs in a container or VM? I'm very interested in that.

[ailispaw]

Oh it seems silly question.. forget it. :)

[ailispaw]

Please provide how to use it with the PR.

[ailispaw]

Are you trying to use GlusterFS from a container or in DockerRoot VM?

[cedvan]

I create 2 DockerRoot VM hosting container GlusterFS Server. And I create 2 DockerRoot VM hosting container GlusterFS Convoy Docker (Client). Convoy need fuse in kernel to share data with server

[ailispaw]

I see. Thanks. So it seems you need just CONFIG_FUSE_FS in kernel.config.

[cedvan]

Ok, I have to create a PR, or you do it ?

[ailispaw]

I can do it, but can not test it with GlusterFS. Please make and test it in your side. I'm looking forward to reading your instruction about GlusterFS with DockerRoot.

[cedvan]

Ok, tell me when I can test, and I'll make a return ;)

[ailispaw]

Do you need docker-root.iso or docker-root.box? I will make it for you.

[cedvan]

I use Vagrant, so need docker-root.box please. Thank you !

[ailispaw]

OK.

[ailispaw]

https://gist.github.com/ailispaw/ec0ebf417ca66283e0de

[ailispaw]

Please download docker-root.box from RAW link at the gist. Or just git clone the gist.

Vagrant.configure(2) do |config|
  config.vm.box = "docker-root-fuse"
  config.vm.box_url = "docker-root.box"
end

$ vagrant up
$ vagrant ssh -c "ls -l /dev/fuse"
crw-------    1 root     root       10, 229 Feb 29 15:59 /dev/fuse

[ailispaw]

Now it's 1 am in Japan. Going to bed. @cedvan See you tomorrow.

[cedvan]

Thank you !

I am in France, I test tonight ;)

Good night man !

[cedvan]

Hi,

I've test. I've other problems to start glusterfs, but /dev/fuse problem is fixed :)

Thank you very much !

[ailispaw]

@cedvan Thank you for testing. Is only CONFIG_FUSE_FS enough for you to use glusterfs with DockerRoot?

[cedvan]

For now , I will return to you if necessary ;)

[ailispaw]

OK, I will wait for your response to merge it.

[ailispaw]

@cedvan Are you still testing? I would like to merge it, if your test is completed successfully.

[cedvan]

hi,

I had test your box with fuse. First problem is solved, thank you !

New error now :

6/3/2016 01:12:40Waiting for metadata.
6/3/2016 01:12:40time="2016-03-06T00:12:40Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/241/ns/mnt -F -- /mnt/sda1/var/lib/docker/overlay/1f2eee3b0e79b3718cec74e0d3938a25615d8db8eaede240a139cea4244f5928/merged/var/lib/rancher/convoy-agent/share-mnt --stage2 /var/lib/rancher/convoy/gluster-convoy-761687de-4ff5-4677-9ce5-d97119df71ac -- /launch volume-agent-glusterfs-internal]"
6/3/2016 01:12:40Timestamp: 2016-03-06 00:12:40.995514645 +0000 UTC
6/3/2016 01:12:40Code: System error
6/3/2016 01:12:40
6/3/2016 01:12:40Message: pivot_root invalid argument
6/3/2016 01:12:40
6/3/2016 01:12:40Frames:
6/3/2016 01:12:40---
6/3/2016 01:12:400: setupRootfs
6/3/2016 01:12:40Package: github.com/opencontainers/runc/libcontainer
6/3/2016 01:12:40File: rootfs_linux.go@69
6/3/2016 01:12:40---
6/3/2016 01:12:401: Init
6/3/2016 01:12:40Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
6/3/2016 01:12:40File: standard_init_linux.go@57
6/3/2016 01:12:40---
6/3/2016 01:12:402: StartInitialization
6/3/2016 01:12:40Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
6/3/2016 01:12:40File: factory_linux.go@242
6/3/2016 01:12:40---
6/3/2016 01:12:403: init·1
6/3/2016 01:12:40Package: main
6/3/2016 01:12:40File: start.go@71
6/3/2016 01:12:40---
6/3/2016 01:12:404: init
6/3/2016 01:12:40Package: main
6/3/2016 01:12:40File: utils.go@177
6/3/2016 01:12:40---
6/3/2016 01:12:405: main
6/3/2016 01:12:40Package: runtime
6/3/2016 01:12:40File: proc.go@58
6/3/2016 01:12:40---
6/3/2016 01:12:41time="2016-03-06T00:12:41Z" level=warning msg="signal: killed"
6/3/2016 01:12:41time="2016-03-06T00:12:41Z" level=error msg="Failed to start (pid 5467) &libcontainer.Process{Args:[]string{\"/launch\", \"volume-agent-glusterfs-internal\"}, Env:[]string{\"CATTLE_CONFIG_URL=https://infra.cedvan.com/v1\", \"HOSTNAME=1f2eee3b0e79\", \"CATTLE_ACCESS_KEY=67D66DFB7166DF758422\", \"CATTLE_AGENT_INSTANCE_AUTH=Basic NjdENjZERkI3MTY2REY3NTg0MjI6NDMySjFweURHbkNBY0FDckttNDJUQlh5NFc5QWNLQWtvVWo4WEdHRg==\", \"CATTLE_SECRET_KEY=432J1pyDGnCAcACrKm42TBXy4W9AcKAkoUj8XGGF\", \"PATH=/var/lib/rancher/convoy-agent:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\", \"PWD=/\", \"SHLVL=1\", \"HOME=/root\", \"CATTLE_URL=https://infra.cedvan.com/v1\", \"_=/var/lib/rancher/convoy-agent/share-mnt\"}, User:\"0:0\", Cwd:\"\", Stdin:(*os.File)(0xc208050028), Stdout:(*os.File)(0xc2080500a0), Stderr:(*os.File)(0xc2080500b0), ExtraFiles:[]*os.File(nil), consolePath:\"\", Capabilities:[]string(nil), ops:libcontainer.processOperations(nil)}: [8] System error: pivot_root invalid argument"
6/3/2016 01:12:41time="2016-03-06T00:12:41Z" level=fatal msg="[8] System error: pivot_root invalid argument"
6/3/2016 01:12:416: goexit

An idea ?

[ailispaw]

What is this an output of? Could you provide your Vagrantfile and environment?

[cedvan]

Output is logs of gluster-convoy from rancher server.

I launch VMs on Ubuntu server 14.04.

My Vagrantfile :

$number_of_nodes = 3
$vm_cpus = 2
$vm_mem = 4096
$vm_gui = false

Vagrant.configure(2) do |config|

  config.vm.box = "ailispaw/docker-root-fuse"
  config.vm.box_url = "docker-root.box"

  (1..$number_of_nodes).each do |i|
    hostname = "rancher-node-%d" % i

    config.vm.define hostname do |node|

        node.vm.hostname = hostname

        node.vm.provider "virtualbox" do |vb|
        vb.cpus = $vm_cpus
            vb.memory = $vm_mem
            vb.gui = $vm_gui
        end

        ip = "192.168.1.2#{i}"
        node.vm.network "public_network", ip: ip, bridge: "eth0"

        node.vm.provision "shell",
        inline: "echo nameserver 192.168.1.2 > /etc/resolv.conf"

        node.vm.provision "shell",
            inline: "echo nameserver 8.8.8.8 >> /etc/resolv.conf"

        # Enable IPVS
        node.vm.provision "shell",
            inline: "sudo modprobe -a ip_vs.ko"

        # Data rancher
        node.vm.synced_folder "/data/nodes/node#{i}", "/var/lib/rancher", disabled: false

        node.vm.synced_folder ".", "/vagrant", disabled: true
    end
  end
end

[ailispaw]

Thanks. I will try it.

What is /var/lib/rancher for? Any other provisions to the VMs for gluster-convoy and/or rancher server? I'm not familiar with them though.

[cedvan]

/var/lib/rancher is use by rancher-agent to save VM state (containers informations, etc...). I mount this folder to preserve informations when reboot VM.

The provisioning depends on the services that we want to start on the VM.

Examples :

• ipvs to keepalived
• fuse to glusterfs, maybe other ?

[cedvan]

I need glusterfs because nfs and vboxsf are not sufficient. My server provide a raid 6 storage. I want use him to save docker volumes is my VMs. With nfs or vboxsf, I've problem with permissions, some containers need to change ownership on volume. I hope that glusterfs will allow...

[cedvan]

At the beginning I wanted to use Rancher OS, but a lot of network problems with vagrant because the docker system. Your box allowed me to get a VM light and clean (network, hostname, dns) to launch my containers. It's great !

Now, only one last missing, the management of the storage volumes. Glusterfs may be the solution ?

[ailispaw]

I see about the problem, but I don't understand your system yet. Could you make a simplified whole example Vagrantfile to use glusterfs for me without others like rancher?

[ailispaw]

Now, only one last missing, the management of the storage volumes. Glusterfs may be the solution ?

I'm considering virtfs (9p). #15 , #20

[ailispaw]

I have to go. See you later.

[cedvan]

I'll analyze virtfs, thank you !

See you later ;)

[ailispaw]

FYI: MariaDB container seems to need 9p2000.L, because it uses fstatvfs().

[cedvan]

Yes, that's why I think I need glusterfs :).

[cedvan]

Possible install glusterfs directly on the VM docker-root ?

[ailispaw]

You can use https://github.com/ailispaw/docker-root-pkg/tree/master/extra/package as an extra package like criu and ipvsadm.

[cedvan]

I am afraid of not being competent enough to succeed, but I will try