Need a way to only invalidate new/changed/removed objects

[kklem0]

CloudFont is a bit slow for the first request after invalidation on each object. It looks like we can get the list from these places, or is there a way to do it already?

https://github.com/jariz/gatsby-plugin-s3/blob/7d26c8d1e24b7e7cd7aabfc12523dd28710cbe1b/src/bin.ts#L218 https://github.com/jariz/gatsby-plugin-s3/blob/7d26c8d1e24b7e7cd7aabfc12523dd28710cbe1b/src/bin.ts#L270 https://github.com/jariz/gatsby-plugin-s3/blob/7d26c8d1e24b7e7cd7aabfc12523dd28710cbe1b/src/bin.ts#L304

This might be related to #68 I'm also wondering how much delay per request we're talking about if I just remove invalidation and add TTL to CloudFront.

• need to invalidate new objects because the 404 or 302 might be cached.
• if invalidating index.html, need to invalidate parent folder with and without /. For example if /a/b/index.html is changed we should invalidate /a/b and /a/b/ and /a/b/index.html.

[YoshiWalsh]

The delay for a cache-miss should always be the same, whether the cache miss was caused by an invalidation or an expired TTL. The other issue with relying on a TTL is that different assets might get cached at different times, so after an update some will expire before others and so viewers could get mismatched assets.

Do you have metrics on the delay for the first request after invalidation? I've never noticed this, S3 seems to be a fast origin.

Selectively invalidating only changed/removed objects does seem elegant though, I think it's definitely something that could be included with #68.

[kklem0]

I think it depends where the object is pulled from, for example our s3 is in US East region but when loading from a bit further location we see spikes once in awhile like Sydney(~500ms)/Hong Kong(~400ms)/Dubai(~400ms)/Germany(~50ms) etc, it's even worse when the objects are bigger, our hope is that if we don't invalidate the needed objects, it won't need to re-fetch from US. And the thought of TTL is that when we invalidate objects it will always need to re-fetch from origin while changing TTL only check if file has changed, but I agree with you that it is not wise to do so.

[YoshiWalsh]

I've just looked into it and it looks like there might be a difference between invalidated objects and TTL-expired objects. If an object is expired, CloudFront uses If-None-Match and If-Modified-Since conditional requests to check if it has the latest version of the object. For larger unmodified objects this could save some time. But for the reasons above I still wouldn't recommend it.