search

scala / scala-jenkins-infra

A Chef cookbook that manages Scala's CI infrastructure.
https://scala-ci.typesafe.com
Apache License 2.0
14 stars 17 forks source link

jenkins-master is slowly running out of disk space #167

Closed SethTisue closed 7 years ago

SethTisue commented 8 years ago

6.8G remains and the growth is gradual, but it won't be much longer

/var/lib/jenkins/jobs/scala-2.11.x-integrate-ide is using 41G (of the 87G total current usage) so that seems like the main culprit

additionally there are 1215 subdirectories under /var/lib/jenkins/jobs/dotty-master-validate-partest/builds, and 308 more under dotty-master-validate-partest-bootstrapped. I tried to do:

% cd /var/lib/jenkins/jobs
% du -sh dotty*partest*

but after 10 minutes I got tired of waiting. but recent builds are taking 27M each, so 1215 times 27M equals ~30G, so I think that's our #2 culprit

adriaanm commented 8 years ago
SethTisue commented 8 years ago

27G available currently after:

adriaan [3:23 AM]
i "permanently archived" a bunch of old scala-ide logs, so that we should be good for a while in the disk space department
[3:23] we should really make logging for that job a bit more concise

adriaanm commented 8 years ago

the real problem was that that partition had run out of inodes, because the dotty partest jobs archive 5000 files per job

I think we need something like https://wiki.jenkins-ci.org/display/JENKINS/Compress+Artifacts+Plugin, or perhaps hardcode it in the job to tar most of the files. (See thread on https://gitter.im/lampepfl/dotty just now)

SethTisue commented 8 years ago

today @adriaanm suggested we stop archiving (or stop archiving so many) the scala-2.11.x-integrate-ide jobs — they are large (69M each, nearly all of it in the one big org.scalaide.TestsSuite-output.txt file) and nobody looks at them. at most we might look at one recent failure

SethTisue commented 8 years ago

not pressing — I check the space on all the Jenkins nodes every 4 days and manually blow away caches as needed

SethTisue commented 7 years ago

the worst problems have been fixed; addressing every space leak isn't worth the effort. there is now doc at https://github.com/scala/scala-jenkins-infra/blob/master/doc/maintenance.md about where to look when space starts running low