mcuadros / gce-docker

Google Cloud Engine integration for Docker
MIT License
59 stars 13 forks source link

More logging information? #6

Open untoldone opened 8 years ago

untoldone commented 8 years ago

I'm currently using this with Rancher. With Rancher, I see log entries like:

2016-12-02 17:45:44,458 ERROR [:] [] [] [] [ecutorService-2] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [compute.instance.remove.reply;agent=341]: Error response from daemon: Unable to remove filesystem for 37b69759385e175622c2f06345db8e53cad042e9ffb8471ee8b1887707e2f13a: remove /var/lib/docker/containers/37b69759385e175622c2f06345db8e53cad042e9ffb8471ee8b1887707e2f13a/shm: device or resource busy

I'm assuming the part Unable to remove filesystem ... is from gce-docker but don't know for sure. Given I have a fleet of machines I'm using this for, its really hard to figure out what host / container / volume this is related to. If it included just the name of the GCE volume, that would solve my problem debugging what is going on.

adamgoose commented 7 years ago

I've been seeing some of this going on as well. It looks like the volume driver is trying to delete disks that it doesn't recognize as volumes, namely my VM boot disks! >< I recently lost a very important volume due to this. I'm not sure if the command is being issued by Rancher, Docker, or this plugin.

5/29/2017 10:33:46 PMt=2017-05-30T03:33:46+0000 lvl=info msg="disk removed" disk=my-cesam elapsed=2.690921065s
5/29/2017 10:33:47 PMt=2017-05-30T03:33:47+0000 lvl=eror msg="request failed" error="googleapi: Error 400: The disk resource 'projects/bridgeconx/zones/us-central1-b/disks/neally-mysql' is already being used by 'projects/bridgeconx/zones/us-central1-b/instances/rancher-host-5', resourceInUseByAnotherResource"

You can see the successful disk removal, followed by the failure. The failure is completely clogging the volume driver container logs. wtf!?

adamgoose commented 7 years ago

light bulb My issue is coming from this: https://github.com/meltwater/docker-cleanup