Find a place to keep knowledge about why Venture Jenkins runs out of disk space

[axch]

Success condition for this task:

• Whoever next has to deal with Jenkins running out of disk space need not rediscover what I now know about this recurring problem.
• Therefore, the knowledge has to be stored somewhere that will be salient when said disk space shortage is discovered
• (by the way, we do not have effective automated monitoring of Jenkins's disk space)

Large things, in order of having been encountered:

• The Jenkins directory grows, partly because Jenkins keeps the build history.
  • Current resolution is to keep it in the 200GB /scratch, which is now 50% full
• The Docker directory grows, because the installation testing builds create Docker containers (and images?)
  • Current resolution is the clean-tmp job, which deletes unused Docker containers daily
  • Could also relocate Docker's container and image store to /scratch
• /tmp grows, because various regular processes create temporary files and do not reliably delete them
  • The files and subdirectories are each not very large, but when I first found this there were 40,000 of them (many being empty directories owned by the jenkins user)
  • Current resolution is the clean-tmp job, which tries to delete junk in /tmp daily

Why not work harder to make jobs clean up after themselves? At the time, I decided that it is valuable to be able to examine created temporaries to debug failures.

[axch]

Relocated the Docker containers to /scratch. Also, need to update the clean-tmp job to actually delete old Docker images, even though they are tagged now.

[axch]

This problem occurred again October 11, 2016. This time the major disk space user was a MongoDB journal in /var/lib/mongodb. This was apparently set up by Grem to store incoming captured BayesDB sessions.

Do we still need to capture BayesDB sessions that way?

• The main way we are distributing the software now is by running IPython over it as a service, which uses a different session capture mechanism, so this might be dead.
• Grem's dmg file would use the mongo install for session capture.
• Any stale instances of bayeslite that others may still have might, as well.
• Rax says this mechanism is not in intentional use.

Options considered:

• Move the journal to /scratch but keep the service running
• Move the journal to an archival location and shut the service down
• Delete the journal and shut the service down

Decided to get disk space by emptying apt-get's package cache instead of dealing with this and forget about this problem until that cache refills, on the grounds that at that point it is likely to be more obvious whether the Mongo install is needed.