CentOS / container-pipeline-service

Code, infrastructure and deployment backend for the CentOS Container Pipeline backing up build system for registry.centos.org
https://registry.centos.org
GNU General Public License v3.0
49 stars 27 forks source link

Enabling node gc for pods, images and containers. #643

Closed mohammedzee1000 closed 6 years ago

mohammedzee1000 commented 6 years ago

This PR adds docs to enable gc on cluster based on kubelet parameters.

Note: This branch was also placeholder for testing out various approaches, and was hard reset when appropriate approach was selected.

Signed-off-by: Mohammed Zeeshan Ahmed mohammed.zee1000@gmail.com

mohammedzee1000 commented 6 years ago

Operations in action:

LIMIT on images :

openshift_node_kubelet_args={'max-pods': ['150'], 'resolv-conf': ['/etc/resolv.conf'],  'image-gc-high-threshold': ['65'], 'image-gc-low-threshold': ['60']}

Basically, if disk usage crosses 65% GC will come into affect. Note only storage affecting docker is taken into account so if volume is mouted on /var then
only that volumes size is taken into account.

Before cleanup is triggered:

$ docker images -f dangling=true
REPOSITORY                                                TAG                 IMAGE ID            CREATED             SIZE
<none>                                                    <none>              4967a565b013        3 minutes ago       200 MB
<none>                                                    <none>              b7e028b3d46f        11 minutes ago      200 MB
registry.centos.org/pipeline-images/ccp-openshift-slave   <none>              cebfdbb54c6d        26 hours ago        1.28 GB

$ df -h
...
/dev/vdb         59G   28G   29G  50% /var
...

Filling up the space to trigger gc:

Multiple instances of :

$ dd if=/dev/zero of=filex.txt count=104857600 bs=1024

space fill:

$ df -h

/dev/vdb         59G   39G   18G  70% /var

After space fill:

$ docker images -f dangling=true
REPOSITORY                                                TAG                 IMAGE ID            CREATED             SIZE
registry.centos.org/pipeline-images/ccp-openshift-slave   <none>              cebfdbb54c6d        26 hours ago        1.28 GB

Note some images, such as this may not be reoved immediately, if the hash is being used somehere, ebven if its dangling
mohammedzee1000 commented 6 years ago

@dharmit updated links

bamachrn commented 6 years ago

LGTM merging

dharmit commented 6 years ago

Why wasn't this merged? Merging now.