search

NVIDIA / deepops

Tools for building GPU clusters
BSD 3-Clause "New" or "Revised" License
1.25k stars 326 forks source link

/var/lib/docker is full #1226

Closed pankajsahtech closed 1 year ago

pankajsahtech commented 2 years ago

We are running kubernetes through deepops with 3 master & in one master /var partition is almost full. /var/lib/docker utilization is more. docker container running for kubernetes & users job training models.

Is it safe to clear space using docker prune volume & docker prune images.

Is any effect over kubernetes functionality post above command

ajdecon commented 2 years ago

Kubernetes includes its own built-in garbage collection functionality which can prune unused images and containers. You can find information on this garbage collection, and how to configure its thresholds, in the Kubernetes documentation: https://kubernetes.io/docs/concepts/architecture/garbage-collection/#containers-images

The docs recommend against running external garbage collection (e.g. docker prune) to avoid interfering with the kubelet's GC.

There's one big exception, though: Kubernetes will only remove images and containers that it launched itself. If you are also running Docker containers on the same nodes but outside of Kubernetes, you will need to manually clean up those containers and images.

pankajsahtech commented 2 years ago

Thanks for update. How much data/volume size ocupied by kubernetes pods to run. any idea checking size

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 60 days with no activity. Please update the issue or it will be closed in 7 days.