search

scala / scala-jenkins-infra

A Chef cookbook that manages Scala's CI infrastructure.
https://scala-ci.typesafe.com
Apache License 2.0
14 stars 17 forks source link

disk space on the behemoths is tight #209

Closed SethTisue closed 6 years ago

SethTisue commented 7 years ago

we keep running out of space. it's possible to keep up with it (as doc'ed), but it's annoying

the root cause is that we have three active development branches and four community builds now, but only the same amount of disk space on the behemoths as when we had two and two, respectively

@adriaanm says of cranking up the disk space:

It's possible, but involves redoing the workers from scratch

which is a pain, so just limping along a couple more months, until we can shut the 2.11 stuff down, is certainly an option.

SethTisue commented 6 years ago

https://github.com/scala/community-builds/commit/81a44e7c137e0be522aee4f5b33ab470445986fb has been a huge help

a fresh community build run, where everything must be rebuilt because the Scala version changed, occasionally still runs out of space (and may incidentally cause a PR validation job to run out of space too, if one happens to be running at the same time)

...but when it happens the problem is transient because a bunch of files get deleted after the failure and then the next run is fine

SethTisue commented 6 years ago

bump. @adriaanm is working on overhauling the Jenkins config so now is a good time to tackle this.

I don't care whether it's more behemoths or bigger behemoths, either way is fine. (at the team meeting just now, Adriaan said he thinks bigger behemoths would be easier.)

there is more and more pressure to have more space:

adriaanm commented 6 years ago

Increased to 400gb On Tue, Feb 6, 2018 at 22:12 Seth Tisue notifications@github.com wrote:

bump. @adriaanm https://github.com/adriaanm is working on overhauling the Jenkins config so now is a good time to tackle this.

I don't care whether it's more behemoths or bigger behemoths, either way is fine. (at the team meeting just now, Adriaan said he thinks bigger behemoths would be easier.)

there is more and more pressure to have more space:

  • the 2.12 build is hugely bigger than it used to be
  • we're up to four builds now: 2.11, 2.11 JDK 8, 2.12, 2.13
  • plus there is a nascent JDK 9 community build we want to expand and start running regularly
  • I would reallllllly loooooove to have space to split the community builds into "stable" and "bleeding edge" versions. the stable ones would stay frozen most of the time, and would be the ones included in the release-main jobs. the bleeding edge jobs would be the crazy-town ones where everything is continuously changing

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/scala/scala-jenkins-infra/issues/209#issuecomment-363565960, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFjy4C1nU8IOxjbNdUOt3Xn2OY-zK3mks5tSMAfgaJpZM4LTR-- .

SethTisue commented 6 years ago

and we have THREE MIGHTY BEHEMOTHS now.