search

worstcase / blockade

Docker-based utility for testing network failures and partitions in distributed applications
http://blockade.readthedocs.org
Apache License 2.0
907 stars 72 forks source link

blockade destroy throwing errors #8

Closed yanfali closed 9 years ago

yanfali commented 9 years ago

Using the examples in read the docs, I get errors while trying to run blockade log and blockade destroy. Does blockade have a specific version of docker and lxc it is targetting?

I'm on an arch linux box with python-3.4.2, lxc-1.0.7 and docker-1.4.1. I installed blockade via pip. The machine has lots of memory and cores 24 (real cores) and 384G of RAM. Let me know if you need more info on my setup.

[root@ev003 blockade]# blockade up
NODE            CONTAINER ID    STATUS  IP              NETWORK    PARTITION
c1              244e48ff6d4f    UP      172.17.0.5      NORMAL
c2              2d74ac31fdeb    UP      172.17.0.6      NORMAL
c3              3fc016b723af    UP      172.17.0.7      NORMAL
[root@ev003 blockade]# blockade status
NODE            CONTAINER ID    STATUS  IP              NETWORK    PARTITION
c1              244e48ff6d4f    UP      172.17.0.5      NORMAL
c2              2d74ac31fdeb    UP      172.17.0.6      NORMAL
c3              3fc016b723af    UP      172.17.0.7      NORMAL
[root@ev003 blockade]# blockade destroy

Unexpected error! This may be a Blockade bug.

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/docker/client.py", line 92, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.4/site-packages/requests/models.py", line 831, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/blockade/cli.py", line 242, in main
    opts.func(opts)
  File "/usr/lib/python3.4/site-packages/blockade/cli.py", line 118, in cmd_destroy
    b.destroy()
  File "/usr/lib/python3.4/site-packages/blockade/core.py", line 118, in destroy
    self.docker_client.remove_container(container_id)
  File "/usr/lib/python3.4/site-packages/docker/client.py", line 886, in remove_container
    self._raise_for_status(res)
  File "/usr/lib/python3.4/site-packages/docker/client.py", line 94, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Cannot destroy container 244e48ff6d4f19e82adfc68e26dd4fd072cd748b1b30ba986bce7e2f193c84a8: Driver devicemapper failed to remove root filesystem 244e48ff6d4f19e82adfc68e26dd4fd072cd748b1b30ba986bce7e2f193c84a8: Device is Busy'")
labisso commented 9 years ago

Thanks for the report. I haven't had a chance to reproduce this issue yet, but it sounds suspiciously like some known issues with docker/devicemapper discussed in this issue: https://github.com/docker/docker/issues/5684#issuecomment-69052334

the linked page here provides more detail and a possible workaround: http://blog.hashbangbash.com/2014/11/docker-devicemapper-fix-for-device-or-resource-busy-ebusy/

You might also try an alternate storage driver, specified with -S to your docker daemon runtime.

I'm working on getting a new blockade release out soon. If I can reproduce this problem, I'll see if I can figure out any workaround within blockade itself.

yanfali commented 9 years ago

Thanks @labisso I'll give it a shot on Monday.

yanfali commented 9 years ago

Using the workaround in the blog article seems to help. Closing this for now. Thank you @labisso for the timely assist :)