Plugin freeze with nonexistent gluster volume

[gregopenit]

First of all, thank you for providing and maintaining this plugin for glusterfs. It is working great but unfortunately I discovered an issue which can be pretty annoying in production.

Describe the bug The plugin freeze if a nonexistent gluster volume is specified. Invalid docker volume cannot be removed nor inspected. A docker restart is required. Force disable the plugin and enable it corrects the problem too.

The plugin is installed and configured like this:

docker plugin install --alias ts-gfs-prod mochoa/glusterfs-volume-plugin --grant-all-permissions --disable docker plugin set ts-gfs-prod SERVERS=removedserver1,removedserver2 docker plugin set ts-gfs-prod SECURE_MANAGEMENT=yes docker plugin enable ts-gfs-prod

To Reproduce I have a gluster volume named containers1-digidev-data with a file file1 containing the string 'volume mounted'.

Working test

$ docker volume create -d ts-gfs-prod containers1-digidev-data/foobar containers1-digidev-data/foobar $ docker run -it --rm -v containers1-digidev-data/foobar:/mnt alpine / # cat /mnt/file1 volume mounted $ docker volume rm containers1-digidev-data/foobar containers1-digidev-data/foobar

Failing test

$ docker volume create -d ts-gfs-prod containers1-digidev-data-invalid/foobar containers1-digidev-data-invalid/foobar $ docker run -it --rm -v containers1-digidev-data-invalid/foobar:/mnt alpine docker: Error response from daemon: Post http://%2Frun%2Fdocker%2Fplugins%2F67f9742d15bf7d90d0ab71cd0243d5b202d0cb9a7f524a13ba71915c8ed31bbe%2Fgfs.sock/VolumeDriver.Mount: net/http: request canceled (Client.Timeout exceeded while awaiting headers). See 'docker run --help'.

Plugin does not respond anymore

I can't remove the failing volume.

$ docker volume rm containers1-digidev-data-invalid/foobar Error response from daemon: get containers1-digidev-data-invalid/foobar: error while checking if volume "containers1-digidev-data-invalid/foobar" exists in driver "ts-gfs-prod:latest": Post http://%2Frun%2Fdocker%2Fplugins%2F67f9742d15bf7d90d0ab71cd0243d5b202d0cb9a7f524a13ba71915c8ed31bbe%2Fgfs.sock/VolumeDriver.Get: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The plugin stops working for any existing volume. I can't inspect any other volume managed by this plugin.

By disabling and renabling the plugin, I am able to remove the failing volume: $ docker plugin disable -f ts-gfs-prod $ docker plugin enable ts-gfs-prod

$ docker volume rm containers1-digidev-data-invalid/foobar containers1-digidev-data-invalid/foobar

Expected behavior An error should be handled like it is for an unexisting folder:

$ docker run -it --rm -v containers1-digidev-data/foobar-invalid:/mnt alpine docker: Error response from daemon: VolumeDriver.Mount: error mounting containers1-digidev-data/foobar-invalid: df: /var/lib/docker-volumes/b940aae4af507f6555698f19d91087116742379bde13f0bcd3f13b78fc384884: Transport endpoint is not connected

fstype should be glusterfs. $ docker volume rm containers1-digidev-data/foobar-invalid containers1-digidev-data/foobar-invalid

Server (please complete the following information):

• Docker host OS: CentOS 7.9
• Docker version: 19.03.13
• GlusterFS server version: glusterfs 8.4
• GlusterFS cluster op-code: 60000 (version 6)