chaosblade-io / chaosblade

An easy to use and powerful chaos engineering experiment toolkit.(阿里巴巴开源的一款简单易用、功能强大的混沌实验注入工具)
https://chaosblade.io
Apache License 2.0
6k stars 949 forks source link

Unable to Create Docker Experiment #605

Open ramgiteng opened 3 years ago

ramgiteng commented 3 years ago

## Issue Description As per instructions, I successfully pulled docker.io/chaosbladeio/chaosblade-demo and ran docker run --it --privileged chaosbladeio/chaosblade-demo Unable to inject Chaos in Docker containers running on host system by running the command: blade create docker cpu fullload --container

Describe what happened (or what feature you want)

Returns output: {"code":502,"success":false,"error":"/bin/sh: docker: not found\n exit status 127"}

Describe what you expected to happen

Expected: Chaosblade should apply 100% cpu to the container running in Docker on my host machine

How to reproduce it (as minimally and precisely as possible)

  1. Run: docker run -it --privileged chaosbladeio/chaosblade-demo
  2. Inside chaosblade container, Run: blade create docker cpu fullload --container containerID

Tell us your environment

OS Version: CentOS 7.5 Docker Version: 1.13.1

Anything else we need to know?

My Docker engine has 5 containers running and I would like to use Chaosblade to inject chaos in those containers. Firstly, I would like to create a docker experiment to apply full CPU load to one of the running containers using ChaosBlade.

Can you please help me troubleshoot this issue? Please let me know if you need further details. Thanks for the support.

xcaspar commented 3 years ago

Please run blade create docker cpu fullload --container CONTAINERID outside of container. You can run blade c docker cpu fullload -h command to see the use examples.

ramgiteng commented 3 years ago

I will try this out. When you say outside the container, you mean outside the Chaosblade container right? I only pulled ChaosBlade image into Docker. I didn't insert ChaosBlade into any of my running containers.

xcaspar commented 3 years ago

chaosblade-demo image contains chaosblade tool, so you can run blade create docker cpu fullload --container CONTAINERID command to trigger the chaosblade execution in the demo container. If you want to inject chaos in your containers, you must append --chaosblade-release XXX flag in the above command. You can see https://chaosblade-io.gitbook.io/chaosblade-help-zh-cn/blade-create-docker/blade-create-docker-cpu for the details.

ramgiteng commented 3 years ago

Ok thank you. I was able to inject cpu load in my containers. I downloaded chaosblade-tool exe (v1.3.0) and it worked.

Would it be the same command? Or was this command only for the demo tool? docker run -it --privileged chaosbladeio/chaosblade-tool:1.3.0

xcaspar commented 3 years ago

chaosblade-demo image is only used for demo.

ramgiteng commented 3 years ago

Thank you for your help, @xcaspar ! Almost everything works. One other question: I ran this command to run Chaosblade in its own container and it works: 1) To run chaosblade in a container : docker run --name=chaosblade -v "//var/run/docker.sock:/var/run/docker.sock" -d registry.cn-hangzhou.aliyuncs.com/chaosblade/chaosblade-tool:1.3.0

However, when I try to execute memory chaos as in the below command, I am getting an error. Command: docker exec -it chaosblade /bin/sh -c "blade create docker mem load --mem-percent 20 --container-name --timeout 20s" Output: {"code":63062, "success":false,"error":docker exec failed ...... , err: {"code":63063, "success":false, "error":"mount -t tmpfs /opt/chaosblade/bin/burnmem_tmpfs -o size=100% (MISSING): cmd exec failed, err: mount: permission denied exit status 32"}}

But when I try to use docker exec on cpu load, it works. Do you know why mem load doesn't work?

xcaspar commented 3 years ago

Less mount and other command privileges. There are two ways to solve it.

  1. Run docker container with --privileged flag.
  2. Run blade command with --mode ram parameter.
ramgiteng commented 3 years ago

Thank you @xcaspar ! It worked. Just one last question - So I am able to run the below command for my existing container: docker exec -it chaosblade /bin/sh -c "blade create docker--container-name --timeout 20s" (For both mem/cpu in one container)

But when I add a new docker container (for example: node app container), and then when I run docker exec -it chaosblade.....I get this error: {"code":63062, "success":false, "error":"DeployChaosBlade: docker exec failed, err: open /opt/chaosblade-1.3.0.tar.gz: no such file or directory"}

I don't get the error in my old container but when I add a new container and use Chaosblade to inject chaos into the new one, I get the above error. Do you have any suggestions? Thanks!