How to use plugin in swarm mode

[metalicjames]

In regular non-swarm mode I can start and use a container with a glusterfs volume using this plugin. However in swarm/service mode, the container always fails to start. Having created a volume called "test" which is confirmed to work in non-swarm mode, the following fails to start:

   sudo docker service create \
   --name test \
   --mount type=volume,volume-driver=hjdr4plugins/docker-volume-glusterfs,src=test,dst=/test \    
   ubuntu:16.04 \
   /bin/bash

With error message on container creation:

    Error grabbing logs: EOF

What am I doing wrong?

[hjdr4]

Hello,

Can you provide the output of docker volume ls in working/non-working situations ? You should have something like hjdr4plugins/docker-volume-glusterfs:latest test

[metalicjames]

Hmm, yes I see that there now. It seems my actual frustration is that I can create services with the volume attached from the CLI, but from docker-compose files I cannot figure out the correct way to do it.

e.g.

version: "3.2"
services:

  vertcoind:
    image: lukechilds/vertcoind
    volumes:
      - vertcoind_data:/data
    command: -rpcuser=middleware -rpcpassword=middleware -txindex

  vtc-middleware-cpp:
    image: jamesl22/vtc-middleware-cpp
    volumes:
      - vertcoind_data:/data
      - index:/tmp/tempdb

volumes:
  vertcoind_data:
    driver: hjdr4plugins/docker-volume-glusterfs
  index:

[hjdr4]

Let's try to figure out your issue. I've been running your stack with no problem. Please provide : docker info docker plugin inspect hjdr4plugins/docker-volume-glusterfs:latest gluster volume list curl http://docker:docker@localhost:9000/api/1.0/volumes

[metalicjames]

 jameslovejoy1@swarm-worker-1:~$ sudo docker info
Containers: 11
 Running: 3
 Paused: 0
 Stopped: 8
Images: 8
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: gijgc86bpiexcplrv4g015a4m
 Is Manager: true
 ClusterID: kim1c17o2y50oky2wv81lpg4o
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.140.0.3
 Manager Addresses:
  10.132.0.2:2377
  10.140.0.3:2377
  10.142.0.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-38-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.654GiB
Name: swarm-worker-1
ID: IUEH:R2XH:QNKF:BPPE:74HB:GALB:6TIO:4LBU:MMC2:PW4V:RGSI:ICLH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
jameslovejoy1@swarm-worker-1:~$ sudo docker plugin inspect hjdr4plugins/docker-volume-glusterfs:latest
[
    {
        "Config": {
            "Args": {
                "Description": "args",
                "Name": "args",
                "Settable": [
                    "value"
                ],
                "Value": null
            },
            "Description": "GlusterFS volume driver for Docker",
            "DockerVersion": "17.06.0-ce",
            "Documentation": "https://docs.docker.com/engine/extend/plugins/",
            "Entrypoint": [
                "/docker-volume-glusterfs"
            ],
            "Env": null,
            "Interface": {
                "Socket": "glusterfs.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": false,
                "Capabilities": [
                    "CAP_SYS_ADMIN"
                ],
                "Devices": [
                    {
                        "Description": "",
                        "Name": "",
                        "Path": "/dev/fuse",
                        "Settable": null
                    }
                ]
            },
            "Mounts": null,
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/var/lib/docker-volumes/_glusterfs",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:5e84b62265b90df8cf13f39ef04f15d04144ba4884bd6cb127b277059d6f1423"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "60e1d83b1b30b4530dc2316a8967e9444a76aa37374de62fe99f1466f874269e",
        "Name": "hjdr4plugins/docker-volume-glusterfs:latest",
        "PluginReference": "docker.io/hjdr4plugins/docker-volume-glusterfs:latest",
        "Settings": {
            "Args": [
                "-servers=gluster0:gluster1:gluster2",
                "-base=/gluster-storage"
            ],
            "Devices": [
                {
                    "Description": "",
                    "Name": "",
                    "Path": "/dev/fuse",
                    "Settable": null
                }
            ],
            "Env": [],
            "Mounts": []
        }
    }
]
jameslovejoy1@swarm-worker-1:~$ sudo gluster volume list
sudo: gluster: command not found
jameslovejoy1@swarm-worker-1:~$ curl http://docker:docker@localhost:9000/api/1.0/volumes
{"err":"Unable to find resource"}
jameslovejoy1@swarm-worker-1:~$ curl http://docker:docker@gluster0:9000/api/1.0/volumes
{
  "data": [
    {
      "bricks": [
        {
          "hostUuid": "2db42640-8652-4683-9169-bd69e944efd8", 
          "name": "gluster0:/gluster-storage/api_vertcoind_data"
        }, 
        {
          "hostUuid": "40698fd8-6794-41e4-807a-fb092409436d", 
          "name": "gluster1:/gluster-storage/api_vertcoind_data"
        }, 
        {
          "hostUuid": "d51d4dde-974d-442d-9b00-35a3d9252c0d", 
          "name": "gluster2:/gluster-storage/api_vertcoind_data"
        }
      ], 
      "distribute": 3, 
      "name": "api_vertcoind_data", 
      "num_bricks": 3, 
      "options": [
        {
          "name": "performance.readdir-ahead", 
          "value": "on"
        }
      ], 
      "replica": 3, 
      "status": "STARTED", 
      "stripe": 1, 
      "transport": "TCP", 
      "type": "REPLICATE", 
      "uuid": "c5556e0d-080c-449b-a272-74f170f339f3"
    }
  ], 
  "ok": true
}

I don't have gluster installed locally on the swarm nodes, they are elsewhere on the network. I used the args="-servers=gluster0:gluster1:gluster2" flag with docker plugin install to specify that.

I should probably also add that I'm using portainer as a UI.

[chriscowley]

Going to leap to say that I am having the same issue. Volumes from Gluster mount fine with basic containers, but Swarm services will not start.