When a new obuilder job is accepted by ocurrent/ocluster the build function in obuilder_build.ml is called. This checks that there is enough free disk space possibly spawning an asynchronous thread do_prune to prune builds before running Builder.build. do_prune invokes Builder.prune which calls Store.prune from db_store.ml. The prune function queries the database to build a list of items ordered by the least recently used and begins deleting them. The delete operation takes several minutes or longer, however the list of items was evaluated at the start. It is possible that the build job which was started in parallel with do_prune or a subsequent job will use one of the cache steps which is in the list to be pruned.
The workaround proposed here is not perfect. It minimises the chance that this will happen by checking the last used time for each item as it is deleted. We will now have one SQL query per item deleted which is less efficient.
The logging has also been updated to show the start and end of the prune operation:
When a new obuilder job is accepted by ocurrent/ocluster the
buildfunction in obuilder_build.ml is called. This checks that there is enough free disk space possibly spawning an asynchronous threaddo_pruneto prune builds before runningBuilder.build.do_pruneinvokesBuilder.prunewhich callsStore.prunefromdb_store.ml. The prune function queries the database to build a list of items ordered by the least recently used and begins deleting them. The delete operation takes several minutes or longer, however the list of items was evaluated at the start. It is possible that the build job which was started in parallel withdo_pruneor a subsequent job will use one of the cache steps which is in the list to be pruned.The workaround proposed here is not perfect. It minimises the chance that this will happen by checking the last used time for each item as it is deleted. We will now have one SQL query per item deleted which is less efficient.
The logging has also been updated to show the start and end of the prune operation: