ailispaw / docker-root

Deprecated: Lightweight Linux for Docker made with Buildroot, moved to https://github.com/bargees/barge
GNU General Public License v2.0
53 stars 7 forks source link

Add support /dev/fuse #21

Closed cedvan closed 8 years ago

cedvan commented 8 years ago

Hi,

Please possible add support fuse module ?

Thank you

ailispaw commented 8 years ago

Hi @cedvan , Thank you for opening an issue.

How can I add that module and test it? Could you make a PR for it?

cedvan commented 8 years ago

I don't know, but I can try

ailispaw commented 8 years ago

If all you need is fuse in user land, you can install with https://github.com/ailispaw/docker-root-pkg.

$ sudo pkg install libfuse

It will install /usr/bin/fusermount and /usr/lib/libfuse.so*.

If you need any kernel configuration for it, please let me know.

ailispaw commented 8 years ago

May need CONFIG_FUSE_FS in kernel.config, too.

ailispaw commented 8 years ago

https://github.com/libfuse/libfuse

cedvan commented 8 years ago

Thank you !

I had install fuse with sudo pkg install libfuse.

Result :

$ modprobe -l |grep fuse
kernel/drivers/message/fusion/mptbase.ko
kernel/drivers/message/fusion/mptscsih.ko
kernel/drivers/message/fusion/mptspi.ko
kernel/drivers/message/fusion/mptsas.ko
kernel/drivers/message/fusion/mptctl.ko
$ cat /dev/fuse
cat: can't open '/dev/fuse': No such file or directory

Need CONFIG_FUSE_FS to enable fuse ? Where is kernel.config file please ?

ailispaw commented 8 years ago

https://github.com/ailispaw/docker-root/blob/master/configs/kernel.config#L2028

ailispaw commented 8 years ago

sudo pkg install libfuse installs only modules in userspace. You need CONFIG_FUSE_FS for kernel modules and devices.

ailispaw commented 8 years ago
diff --git a/configs/buildroot.config b/configs/buildroot.config
index 2a62e5c..571277b 100644
--- a/configs/buildroot.config
+++ b/configs/buildroot.config
@@ -1065,7 +1065,7 @@ BR2_PACKAGE_OPENSSL=y
 # BR2_PACKAGE_GAMIN is not set
 # BR2_PACKAGE_LIBCONFIG is not set
 # BR2_PACKAGE_LIBCONFUSE is not set
-# BR2_PACKAGE_LIBFUSE is not set
+BR2_PACKAGE_LIBFUSE=y
 # BR2_PACKAGE_LIBLOCKFILE is not set
 # BR2_PACKAGE_LIBNFS is not set
 # BR2_PACKAGE_LIBSYSFS is not set
diff --git a/configs/kernel.config b/configs/kernel.config
index 2fad792..602e5dc 100644
--- a/configs/kernel.config
+++ b/configs/kernel.config
@@ -2025,7 +2025,8 @@ CONFIG_INOTIFY_USER=y
 # CONFIG_QUOTA is not set
 # CONFIG_QUOTACTL is not set
 # CONFIG_AUTOFS4_FS is not set
-# CONFIG_FUSE_FS is not set
+CONFIG_FUSE_FS=y
+# CONFIG_CUSE is not set
 CONFIG_OVERLAY_FS=y

 #
cedvan commented 8 years ago

Ok thank you very much !

I will create a PR as soon as possible ;)

ailispaw commented 8 years ago

If that modification doesn't make kernel bigger significantly, I will add CONFIG_FUSE_FS in the next release.

ailispaw commented 8 years ago

BTW, what fuse will benefit and what is your use case?

cedvan commented 8 years ago

I need fuse to support glusterfs :)

ailispaw commented 8 years ago

Is glusterfs in a container or VM? I'm very interested in that.

ailispaw commented 8 years ago

Oh it seems silly question.. forget it. :)

ailispaw commented 8 years ago

Please provide how to use it with the PR.

ailispaw commented 8 years ago

Are you trying to use GlusterFS from a container or in DockerRoot VM?

cedvan commented 8 years ago

I create 2 DockerRoot VM hosting container GlusterFS Server. And I create 2 DockerRoot VM hosting container GlusterFS Convoy Docker (Client). Convoy need fuse in kernel to share data with server

ailispaw commented 8 years ago

I see. Thanks. So it seems you need just CONFIG_FUSE_FS in kernel.config.

cedvan commented 8 years ago

Ok, I have to create a PR, or you do it ?

ailispaw commented 8 years ago

I can do it, but can not test it with GlusterFS. Please make and test it in your side. I'm looking forward to reading your instruction about GlusterFS with DockerRoot.

cedvan commented 8 years ago

Ok, tell me when I can test, and I'll make a return ;)

ailispaw commented 8 years ago

Do you need docker-root.iso or docker-root.box? I will make it for you.

cedvan commented 8 years ago

I use Vagrant, so need docker-root.box please. Thank you !

ailispaw commented 8 years ago

OK.

ailispaw commented 8 years ago

https://gist.github.com/ailispaw/ec0ebf417ca66283e0de

ailispaw commented 8 years ago

Please download docker-root.box from RAW link at the gist. Or just git clone the gist.

Vagrant.configure(2) do |config|
  config.vm.box = "docker-root-fuse"
  config.vm.box_url = "docker-root.box"
end
$ vagrant up
$ vagrant ssh -c "ls -l /dev/fuse"
crw-------    1 root     root       10, 229 Feb 29 15:59 /dev/fuse
ailispaw commented 8 years ago

Now it's 1 am in Japan. Going to bed. @cedvan See you tomorrow.

cedvan commented 8 years ago

Thank you !

I am in France, I test tonight ;)

Good night man !

cedvan commented 8 years ago

Hi,

I've test. I've other problems to start glusterfs, but /dev/fuse problem is fixed :)

Thank you very much !

ailispaw commented 8 years ago

@cedvan Thank you for testing. Is only CONFIG_FUSE_FS enough for you to use glusterfs with DockerRoot?

cedvan commented 8 years ago

For now , I will return to you if necessary ;)

ailispaw commented 8 years ago

OK, I will wait for your response to merge it.

ailispaw commented 8 years ago

@cedvan Are you still testing? I would like to merge it, if your test is completed successfully.

cedvan commented 8 years ago

hi,

I had test your box with fuse. First problem is solved, thank you !

New error now :

6/3/2016 01:12:40Waiting for metadata.
6/3/2016 01:12:40time="2016-03-06T00:12:40Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/241/ns/mnt -F -- /mnt/sda1/var/lib/docker/overlay/1f2eee3b0e79b3718cec74e0d3938a25615d8db8eaede240a139cea4244f5928/merged/var/lib/rancher/convoy-agent/share-mnt --stage2 /var/lib/rancher/convoy/gluster-convoy-761687de-4ff5-4677-9ce5-d97119df71ac -- /launch volume-agent-glusterfs-internal]"
6/3/2016 01:12:40Timestamp: 2016-03-06 00:12:40.995514645 +0000 UTC
6/3/2016 01:12:40Code: System error
6/3/2016 01:12:40
6/3/2016 01:12:40Message: pivot_root invalid argument
6/3/2016 01:12:40
6/3/2016 01:12:40Frames:
6/3/2016 01:12:40---
6/3/2016 01:12:400: setupRootfs
6/3/2016 01:12:40Package: github.com/opencontainers/runc/libcontainer
6/3/2016 01:12:40File: rootfs_linux.go@69
6/3/2016 01:12:40---
6/3/2016 01:12:401: Init
6/3/2016 01:12:40Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
6/3/2016 01:12:40File: standard_init_linux.go@57
6/3/2016 01:12:40---
6/3/2016 01:12:402: StartInitialization
6/3/2016 01:12:40Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
6/3/2016 01:12:40File: factory_linux.go@242
6/3/2016 01:12:40---
6/3/2016 01:12:403: init·1
6/3/2016 01:12:40Package: main
6/3/2016 01:12:40File: start.go@71
6/3/2016 01:12:40---
6/3/2016 01:12:404: init
6/3/2016 01:12:40Package: main
6/3/2016 01:12:40File: utils.go@177
6/3/2016 01:12:40---
6/3/2016 01:12:405: main
6/3/2016 01:12:40Package: runtime
6/3/2016 01:12:40File: proc.go@58
6/3/2016 01:12:40---
6/3/2016 01:12:41time="2016-03-06T00:12:41Z" level=warning msg="signal: killed"
6/3/2016 01:12:41time="2016-03-06T00:12:41Z" level=error msg="Failed to start (pid 5467) &libcontainer.Process{Args:[]string{\"/launch\", \"volume-agent-glusterfs-internal\"}, Env:[]string{\"CATTLE_CONFIG_URL=https://infra.cedvan.com/v1\", \"HOSTNAME=1f2eee3b0e79\", \"CATTLE_ACCESS_KEY=67D66DFB7166DF758422\", \"CATTLE_AGENT_INSTANCE_AUTH=Basic NjdENjZERkI3MTY2REY3NTg0MjI6NDMySjFweURHbkNBY0FDckttNDJUQlh5NFc5QWNLQWtvVWo4WEdHRg==\", \"CATTLE_SECRET_KEY=432J1pyDGnCAcACrKm42TBXy4W9AcKAkoUj8XGGF\", \"PATH=/var/lib/rancher/convoy-agent:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\", \"PWD=/\", \"SHLVL=1\", \"HOME=/root\", \"CATTLE_URL=https://infra.cedvan.com/v1\", \"_=/var/lib/rancher/convoy-agent/share-mnt\"}, User:\"0:0\", Cwd:\"\", Stdin:(*os.File)(0xc208050028), Stdout:(*os.File)(0xc2080500a0), Stderr:(*os.File)(0xc2080500b0), ExtraFiles:[]*os.File(nil), consolePath:\"\", Capabilities:[]string(nil), ops:libcontainer.processOperations(nil)}: [8] System error: pivot_root invalid argument"
6/3/2016 01:12:41time="2016-03-06T00:12:41Z" level=fatal msg="[8] System error: pivot_root invalid argument"
6/3/2016 01:12:416: goexit

An idea ?

ailispaw commented 8 years ago

What is this an output of? Could you provide your Vagrantfile and environment?

cedvan commented 8 years ago

Output is logs of gluster-convoy from rancher server.

I launch VMs on Ubuntu server 14.04.

My Vagrantfile :

$number_of_nodes = 3
$vm_cpus = 2
$vm_mem = 4096
$vm_gui = false

Vagrant.configure(2) do |config|

  config.vm.box = "ailispaw/docker-root-fuse"
  config.vm.box_url = "docker-root.box"

  (1..$number_of_nodes).each do |i|
    hostname = "rancher-node-%d" % i

    config.vm.define hostname do |node|

        node.vm.hostname = hostname

        node.vm.provider "virtualbox" do |vb|
        vb.cpus = $vm_cpus
            vb.memory = $vm_mem
            vb.gui = $vm_gui
        end

        ip = "192.168.1.2#{i}"
        node.vm.network "public_network", ip: ip, bridge: "eth0"

        node.vm.provision "shell",
        inline: "echo nameserver 192.168.1.2 > /etc/resolv.conf"

        node.vm.provision "shell",
            inline: "echo nameserver 8.8.8.8 >> /etc/resolv.conf"

        # Enable IPVS
        node.vm.provision "shell",
            inline: "sudo modprobe -a ip_vs.ko"

        # Data rancher
        node.vm.synced_folder "/data/nodes/node#{i}", "/var/lib/rancher", disabled: false

        node.vm.synced_folder ".", "/vagrant", disabled: true
    end
  end
end
ailispaw commented 8 years ago

Thanks. I will try it.

What is /var/lib/rancher for? Any other provisions to the VMs for gluster-convoy and/or rancher server? I'm not familiar with them though.

cedvan commented 8 years ago

/var/lib/rancher is use by rancher-agent to save VM state (containers informations, etc...). I mount this folder to preserve informations when reboot VM.

The provisioning depends on the services that we want to start on the VM.

Examples :

cedvan commented 8 years ago

I need glusterfs because nfs and vboxsf are not sufficient. My server provide a raid 6 storage. I want use him to save docker volumes is my VMs. With nfs or vboxsf, I've problem with permissions, some containers need to change ownership on volume. I hope that glusterfs will allow...

cedvan commented 8 years ago

At the beginning I wanted to use Rancher OS, but a lot of network problems with vagrant because the docker system. Your box allowed me to get a VM light and clean (network, hostname, dns) to launch my containers. It's great !

Now, only one last missing, the management of the storage volumes. Glusterfs may be the solution ?

ailispaw commented 8 years ago

I see about the problem, but I don't understand your system yet. Could you make a simplified whole example Vagrantfile to use glusterfs for me without others like rancher?

ailispaw commented 8 years ago

Now, only one last missing, the management of the storage volumes. Glusterfs may be the solution ?

I'm considering virtfs (9p). #15 , #20

ailispaw commented 8 years ago

I have to go. See you later.

cedvan commented 8 years ago

I'll analyze virtfs, thank you !

See you later ;)

ailispaw commented 8 years ago

FYI: MariaDB container seems to need 9p2000.L, because it uses fstatvfs().

cedvan commented 8 years ago

Yes, that's why I think I need glusterfs :).

cedvan commented 8 years ago

Possible install glusterfs directly on the VM docker-root ?

ailispaw commented 8 years ago

You can use https://github.com/ailispaw/docker-root-pkg/tree/master/extra/package as an extra package like criu and ipvsadm.

cedvan commented 8 years ago

I am afraid of not being competent enough to succeed, but I will try