search

ipfs-inactive / dev-team-enablement

[ARCHIVED] Dev Team Enablement Working Group
3 stars 1 forks source link

Cleaning after go-ipfs jobs #70

Open victorb opened 6 years ago

victorb commented 6 years ago

Right now, after having the workers running for a while, they run out of space since the go-ipfs job does not clean up after itself.

From IRC:

[2018-06-04 11:34:06] <daviddias> victorbjelkholm: Mac OS X workers are dying on Jenkins
[2018-06-04 11:34:26] <victorbjelkholm> oh, no goodie
[2018-06-04 11:38:26] <victorbjelkholm> Magik6k_: go-ipfs Jenkinsfile seems to be missing a call to `cleanWs` in the end of the pipeline, could you add it?
[2018-06-04 11:50:27] <Magik6k_> victorbjelkholm: the problem is that calling cleanws in pipeline will make the job about 2x slower. Is there a way to tell jenkins to remove workspace after they aren't used for some time?
[2018-06-04 11:51:10] <victorbjelkholm> hm, probably we should setup a job that once every day removes all inactive workspaces then
[2018-06-04 11:51:43] <Magik6k_> yeah, if that's possible it would be great
[2018-06-04 11:52:07] <victorbjelkholm> hm, why it takes so long time though? js-ipfs cleanWS takes < 1 second, and that's with all the node modules and jazz
[2018-06-04 11:53:32] <Magik6k_> it's gx that needs to re-get all stuff for the builds (build time goes up from 7 to 16min on fresh workspaces)

We need to find a way of caching the gx packages outside of the workspace on the workers. Pinning them on the master and running IPFS on the macOS workers would make the install speed trivial, and the workers not needing to keep the builds around for more than necessary.

  • [ ] Pin all gx packages on jenkins master (already running IPFS)
  • [ ] Install and run go-ipfs on macOS workers
  • [ ] Setup go-ipfs (and golang pipelines in general) to install gx packages from IPFS
  • [ ] Clean workspace after go-ipfs jobs

@magik6k sounds like a reasonable idea?

From @magik6k

getting gx deps tends to be slow even with local node, but https://github.com/whyrusleeping/gx-go/pull/38 should help with that, so yeah, that can work

Moved from https://github.com/ipfs/jenkins-libs/issues/15

victorb commented 6 years ago

@magik6k can you confirm that the cleanup now happens when go projects are done building and testing? Also in case of failures.

magik6k commented 6 years ago

go-ipfs should be cleaning up - https://github.com/ipfs/go-ipfs/blob/master/ci/Jenkinsfile#L95, not sure about other go projects