Add caching of repositories

[Olivia5k]

Repositories that need to be built a lot are usually quite large in size. A lot of time and electricity is wasted in most build systems by cloning the repositories over and over again. As repositories grow, this can add minutes to the total build time. This is silly and easy to overcome.

My suggested clone workflow for agents (repo entity/repo.git used as example):

• Check if $REPOCACHE/entity/repo exists. If it does not, do a bare clone into it. If it does, fetch -a.
• Clone the bare repo into $WORKSPACE.

  Todo

• [ ] Implement the above.
• [ ] Add gitcache configuration field to piperd.yml.
• [ ] Add optional cache size limiting. Once the limit is reached, delete the least used repositories until the size is below the limit again.

  Nice to haves

• [ ] Add support for forks. That way forks of large repositories don't need their own cache entry - they can just use the upstream and apply the patches from the fork.

[Olivia5k]

@daenney: What do you thinks?

[daenney]

My only concern is one of tainting the environment. I have a habit of wanting to run tests in full isolation, creating and destroying the entity that runs/ran the tests as needed.

I like this idea but it would require a stateful agent that holds on to the repo and we clone from there. I wouldn't want this to be the same agent that runs my tests. Having the agent that runs the tests hold on to the repo would also be problematic if we implement support for things like containers.

[Olivia5k]

I am usually in agreement, but when one can save multiple minutes of build time I quickly let that value erode a bit.

I've thought about the container part as well. Couldn't the repository caching just be outside of the containers and the directories needed are linked into the containers?

[daenney]

Yes certainly could. Would make the build agents more permanent "fixtures" in that sense but as long as we isolate the environment in which the tests run from the agent environment itself I don't see any issues.

If people have a need to continuously spin up and down the agents they'd simply lose the caching capability.

[Olivia5k]

Definitely! And this will not infer the capability of recycling agents - that would just mean that the first builds after that would take a bit longer due to cloning!