search

ipfs-inactive / dev-team-enablement

[ARCHIVED] Dev Team Enablement Working Group
3 stars 1 forks source link

Linux worker out of disk space #118

Closed victorb closed 6 years ago

victorb commented 6 years ago

From @vmx on August 7, 2018 23:36

CI currently fails beause a (some?) Linux worker run out of disk space. See https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs-ipfs-unixfs-engine/detail/PR-223/8/pipeline/22 (and previous build of this job) for more.

Copied from original issue: ipfs/jenkins#122

victorb commented 6 years ago

From @Stebalien on August 11, 2018 20:43

@VictorBjelkholm is there anything the go team can do to get this resolved?

victorb commented 6 years ago

From @magik6k on August 12, 2018 13:47

There is a fix waiting in https://github.com/ipfs/go-ipfs/pull/5053/files#diff-b7ff5824046cf0b1acd30703f283738dR65, but it sort-of depends on js-api tests being stable. I'll open a separate pr for that

victorb commented 6 years ago

From @magik6k on August 12, 2018 13:52

(https://github.com/ipfs/go-ipfs/pull/5374)

victorb commented 6 years ago

Websites are now properly cleaning up after themselves.

We're currently missing golang builds to clean their workspace + cleaning up docker volumes (containers should already be cleaned up now). Otherwise the space is mostly used for golang projects + yarn cache.

Stebalien commented 6 years ago

Shouldn't jenkins be cleaning up after builds?

victorb commented 6 years ago

@Stebalien by default, Jenkins keeps the workspaces across builds.

victorb commented 6 years ago

Several mitigations against this now that should help us. Metrics from all workers regarding disk space (and many other metrics) for alerting and job that runs each day to clean-up. Also made the pipelines better to cleanup after themselves, so we should not see this anymore.