Better Buildpack Caching

[onsi]

I've just added

https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md

It's a WIP proposal for improving Diego's staging performance. I'm (very) open to suggestions. We aren't likely to schedule this immediately but wanted to get some thoughts together.

[fraenkel]

With this shared cache, you could end up with more items than you expected? You say I want cache foo and I want this 1 thing in there, but someone else says 2 things for the same cache.

Do we care that the cache may change as you are using it? User A is using cache foo, user B says add/update this thing in the cache.

How do we remove things from the cache? Unless we are replacing a consistent name, the cache will grow and grow. What if a buildpack is removed? How do we know it should be removed from the cache?

How are we managing disk space? Does the cache count against the total disk allowed?

This is exactly how we are managing buildpacks with DEAs. However, I would add one additional caveat. We only "prune" a shared cache if all the users agree on its content. If we have multiple stagers and they have different content, we punt. We don't really know who has the correct list of items when there are changes afoot.

[onsi]

@fraenkel - I thought of a bunch of these issues as I was writing the proposal up and wanted to see if we could make headway as a group.

but maybe there's a simpler way that resolves most of these. I'll sketch it out briefly here and can update the doc if it has merit. I propose:

• we split out DownloadAction from CachedDownloadAction
• CachedDownloadAction looks like {From: "url", To: "/path", CacheKey: "cache-key"}
• for each container, the executor creates a cache directory unique to said container and bindmounts it in as /cached-downloads
• instead of having a shared cache between containers, the executor manages a single global cache where all CachedDownload assets go but then symlinks just the requested files into the container's cached-downloads folder. The /path in the CachedDownloadAction will always be relative to /cached-downloads.

So how does this single global cache work? I think there's a path forward and we can hash out the details as a team but I was thinking something like this might work (apologies for the detail, I'm working through it as I go. I imagine I'm overcomplicating things and there's a much simpler way forward):

• when fetching an item:
  • if it's not in the cache then:
  • download it
  • uncompress/untar/unzip it if necessary
  • put it in /global-cache/{cache-key}-0
  • record (in-memory though we can make the cache more durable if need be) something like {CacheKey: cacheKey, Index: 0, Date: time.Now(), Etag: etag}
  • if it is in the cache:
  • check the etag, if its up-to-date then don't download a new copy
  • if it's not up-to-date then:
    • if the existing cached item is in use, store the new item under /global-cache/{cache-key}-(m+1) where m is the greatest index for this cache-key
    • if the existing cached item is not in use, overwrite the existing entry in /global-cache/{cache-key}-m where m is the greatest index for this cache-key
• if there is no room to fetch the item:
  • attempt to evict unused entries in the cache
  • if that fails: error
• after downloading the item: symlink it into the container's cache and increment the item's usage counter.
• when the container no longer exists, decrement the item's usage counter.
• items with a usage counter of 0 are reaped if:
  • there is an item with a greater cache index
  • the cache is within some threshold of being full

it's complicated. Though, honestly, the existing cache is already doing some of this stuff. We just need to be careful about not overwriting things that might be in use and then just symlink things into the directory that is bind-mounted into the container.

We should probably start by making sure that this whole symlink+bindmount trick will work. If it does then I think this approach is feasible and I don't think it's a terrible API to say that Diego can do cached downloads and its up to the consumer to know that they live in /cached-downloads.

For the concrete case of buildpacks we'd have to teach the builder to look in /cached-downloads for the buildpacks. That shouldn't be terrible and it could even perform a subsequent symlink if it needs to. We should probably spike and make sure the buildpacks actually work if symlinked in like this (I imagine there may be a whole mess of issues around users and permissions, etc...)

Amicrazy?

[fraenkel]

What you are proposing does make sense. You have made assumptions about the content a url delivers, and does align with the lifecycle of the content we tend to download. There are the corner cases which we should just handle as non-cached items like no etag support, etc... It might be interesting to add stats around the cache behavior. It would provide useful feedback to operators to know when their caches are too small.

[onsi]

aight cool -- I'll update the doc and add a story.

[onsi]

closing this in favor of #30