global cache each target

[dana-resonai]

• [x] Create an interface for loading remote cache
• [x] In load_target_from_cache - if the target is not cached locally but is cached remotely, copy the remote cache to the local cache of the target. Then use it instead of building.
• [x] Add a flag that tells whether to upload the local cache to the remote cache (set to False by default, non-robot users will usually run with the default)
• [x] Add test results to the cached objects
• [x] Deal with deleting the cache once in a while (something like 2 weeks old cache)
• [x] implement periodic expunge.
• [ ] Implement optimized (e.g. batched) download / upload. Alternatively, check if Google's remote directory works.

[itamaro]

I believe we will need to put in place the infra to do things like:

• per target, keep track of cached size of artifacts vs. local build time, so we can add logic to optimize fetch-vs-build decision.
• maintain "last used" and "last updated" for cached targets as input for cleanup prioritization.
• add some API layer that will allow batch querying remote cache status of multiple targets to avoid the per-target-overhead.
• maybe even have local ybt service running on dev machines, that maintains consistency between remote cache metadata and local proxy, so local builds usually don't need to wait for network.

[dlehavi-resonai]

@itamaro

• AFAIK optimizing this automatically is far more tricky than build time / size. E.g. fetching a 100 1M files is way harder than fetching a 100M file (something which gsutils tries to optimize, and I'm sure AWS / Azure has similar solutions).
• There is something with remote fetch which is different from local cache: Since we link statically (our own code, not STL), if we need a certain target, and it exists remotely, we don't need to fetch the dependencies. This generalizes to "fetch the highest nodes you can build in the DAG of the set of requested targets"
• We are still not sure on how to implement cleanups. @eyalfink suggested using GCE mechanism for self destructing buckets (I'll let you in on a dark secret - I just erase old stuff from my cache on my local machine)
• Our most pressing (cache related) pain points are:
  1. On jenkins: running tests and building cpp targets remotely over and over again 2: locally: pulling master IIUC, with the ybt service you try to solve 2, but this happens at most once or twice a day, so I'm not sure it's worth keeping a service for