search

ros-infrastructure / buildfarm_deployment

Apache License 2.0
30 stars 39 forks source link

Adjust docker image cleanup tolerances. #238

Open nuclearsandwich opened 4 years ago

nuclearsandwich commented 4 years ago

Jenkins and the cleanup script do their accounting slightly differently so there are times when the cleanup script runs and Jenkins still considers the disk to be too full. By bumping the cleanup minimum to 60G that gives us a pretty comfortable margin over the 50G minimum for Jenkins. I also bumped the percentage so we don't flap on and off as frequently.

These changes affect all agents but I don't think any of them are harmed by the tuning of these parameters.

As we work on the future agents we're going to delegate cleanup responsibility to docker prune commands, which we are already doing for the Jenkins host which previously did not have a configured cleanup script.

dirk-thomas commented 4 years ago

The specific numbers of the different nodes should be taken into consideration here (non-ARM as well as ARM nodes) and based on their disk sizes the limits should be set.

Atm for e.g. ARM nodes which have less than 500 GB disk space the cleanup script only kicks in at 50 GB (since 10% of less than 500 GB is less) but the configured threshold in Jenkins is also 50 GB which leaves no margin. So when the cleanup script kicks in it is already too late and Jenkins will take the node offline.