search

pandastrike / orca

Distributed load testing framework
21 stars 3 forks source link

Examine Docker Container Space Issues #25

Open freeformflow opened 9 years ago

freeformflow commented 9 years ago

The current CoreOS cluster we're using is made of m3.large AWS machines. However, we are getting errors when trying to create container that reference space limitations. We need to figure out if this is a configuration in CoreOS, the Docker Host, or somewhere else because it is unacceptably small.

peterlongnguyen commented 9 years ago

This issue occured while I was setting up the https://github.com/pandastrike/timezone repo. Here's the printout of the error.

Nov 21 21:14:34 ip-10-229-48-43.us-west-1.compute.internal docker[28372]: Pulling repository peterlnguyen/timezone
Nov 21 21:14:59 ip-10-229-48-43.us-west-1.compute.internal systemd[1]: timezone-app@05.service: control process exited, code=exited status=1
Nov 21 21:14:59 ip-10-229-48-43.us-west-1.compute.internal systemd[1]: Failed to start Timezone App for Panda Strike.
Nov 21 21:14:59 ip-10-229-48-43.us-west-1.compute.internal systemd[1]: Unit timezone-app@05.service entered failed state.
Nov 21 21:15:06 ip-10-229-48-43.us-west-1.compute.internal docker[28372]: 2014/11/21 21:14:59 Error pulling image (latest) from peterlnguyen/timezone, open /var/lib/docker/btrfs/subvolumes/414b00c50c0855134766475800f08464537300124372e672fc797f41f671d969/usr/include/glib-2.0/gio/ginetaddressmask.h: no space left on device

I temporarily solved it by docker rm and docker rmi to free up space. Seeing as how this is barely a couple of gigs, the issue is likely some default limit that needs to be increased.

ip-10-229-48-43 docker # docker images

REPOSITORY                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
<none>                      <none>              4b356dae592c        About an hour ago   864 MB
pandapup/orca_agent         latest              f3902ad7eb57        8 days ago          957.6 MB
pandapup/skydns             latest              39b4badffc9a        12 days ago         1.438 GB
pandapup/coreos_reflector   latest              6103cdcb4bd7        2 weeks ago         830.6 MB
dockerfile/redis            latest              806930947ad9        3 weeks ago         432.7 MB

ip-10-229-48-43 docker # docker ps -a

CONTAINER ID        IMAGE                              COMMAND                CREATED             STATUS                     PORTS                NAMES
fdb518c42348        pandapup/skydns:latest             "/bin/bash -c 'sleep   17 minutes ago      Up 17 minutes              0.0.0.0:53->53/tcp   skydns
7e00adfa8a3b        pandapup/coreos_reflector:latest   "/bin/bash -c 'cd co   2 days ago          Exited (-1) 14 hours ago                        reflector-05
8fe1e3cf979c        pandapup/orca_agent:latest         "\"/bin/bash -c 'sou   2 days ago          Exited (0) 2 days ago                           drone-0200
bca16c9765e8        dockerfile/redis:latest            "redis-server /etc/r   2 days ago          Exited (0) 2 days ago                           redis-02
fcabde87cc0a        pandapup/orca_agent:latest         "\"/bin/bash -c 'sou   2 days ago          Exited (1) 2 days ago                           leader-02
peterlongnguyen commented 9 years ago

Issue is duplicated from https://github.com/pandastrike/panda-wiki/issues/59

Please update both as appropriate.

freeformflow commented 9 years ago

@peterlnguyen, I think I found the source of the problem. Docker's host will take up as much space as it can until it runs out of room. What's limiting us is the space allocated to the rootfs partition.

Check out this gist that contains the result of a sudo df -H pull on the CoreOS cluster. Compare the rootfs line to the Docker line.

I saw where I can ask Amazon for more hard drive space, and the good news is that that is cheap. What I don't know is how to allocate that space to rootfs. Do you or Lance know any Linux magic for that?