Open ndelitski opened 8 years ago
Hi @ndelitski
In fact from the log, these request comes from Docker, rather than Convoy.
You see e.g.:
time="2015-10-11T18:54:27Z" level=debug msg="Handle plugin create volume: POST /VolumeDriver.Create" pkg=daemon
This kind of request are from Docker. So it's not trigger by your Convoy command.
I found upon the Docker daemon start, it try to find all the existing volumes with current driver(which you may change to Convoy), and ask it to create the volume if they're not there. You can see Handle plugin create volume: POST /VolumeDriver.Create
in the log, which means it's coming from Docker(Handle plugin xxx
would come from Docker).
@ndelitski
And I am unable to reproduce you mentioned "If volume name contains dots additional volume named . will be created..." I tried the middle or end of name, nothing unusual happens... Could you show example of command?
Thanks!
Hi @yasker You mentioned that docker demon asks a volume driver to create new volumes if they does not exist. I wonder why convoy created new volumes with random names(11 or 12 - maximum that can be attached to EC2 host) and not complaining about volume not exists? As for dots i just noticed in AWS Console a volume with name '.' and thought it might be related to the name. If i encounter this error i again i will come with more details. In a case above i attached volumes with names vol-rancher.cattle, vol-conf.redis, vol-conf-redis. Docker 1.8.2
AMI ID ubuntu-trusty-14.04-amd64-server-20150325 (ami-47a23a30)
ubuntu ~> docker version
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Wed Oct 7 17:48:28 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Wed Oct 7 17:48:28 UTC 2015
OS/Arch: linux/amd64
ubuntu ~> docker info
Containers: 8
Images: 174
Storage Driver: devicemapper
Pool Name: docker-202:1-526147-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 5.728 GB
Data Space Total: 107.4 GB
Data Space Available: 66.46 GB
Metadata Space Used: 9.839 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.138 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-48-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 3.676 GiB
Name: ip-172-30-1-166
ID: S3VY:3PTC:TIGU:ICDC:YFXG:NFS4:ZZXO:CICW:NXXB:VKRP:M6O2:HI4R
WARNING: No swap limit support
@ndelitski
In fact Docker send out create volume request and didn't check if volume exists or not. It would expect driver to provide the volume, identified by the name, and didn't provide information that it would expect volume exists or not. So there is no way for Convoy to know whether Docker want to create a new volume or use a existing volume, thus it would only comply with the request and create the volume.
I guess the name is not random. The volume names should come from your old Docker containers. You can run docker inspect <container_name>
to see how container's volume allocated and what's the name of it. And probably you still have the volumes with the same name at /var/lib/docker/volumes
locally.
I am seeing this in the following scenario:
I think this is due to Convoy not responding to the POST /VolumeDriver.Get
request from Docker:
time="2016-05-03T12:26:28Z" level=debug msg="Handle plugin activate: POST /Plugin.Activate" pkg=daemon
time="2016-05-03T12:26:28Z" level=debug msg="Response: {\n\t\"Implements\": [\n\t\t\"VolumeDriver\"\n\t]\n}" pkg=daemon
time="2016-05-03T12:26:28Z" level=error msg="Handler not found: POST /VolumeDriver.Get" pkg=daemon
time="2016-05-03T12:26:28Z" level=error msg="Handler not found: POST /VolumeDriver.Get" pkg=daemon
time="2016-05-03T12:26:28Z" level=debug msg="Handle plugin create volume: POST /VolumeDriver.Create" pkg=daemon
time="2016-05-03T12:26:28Z" level=debug msg="Request from docker: &{local_volume map[]}" pkg=daemon
time="2016-05-03T12:26:28Z" level=debug msg="Create a new volume local_volume for docker" pkg=daemon
Just run
sudo convoy create vol-rancher-cattle
and got many ebs volumes created in 10 mins. Killed host, stated new one and issue occured again... If volume name contains dots additional volume named.
will be created... Convoy 0.3.0 Daemon in EBS mode daemon logs:fdisk -l