Fastly cache invalidation is either partial or not working

[mensfeld]

Creating this based on the slack thread:

https://bundler.slack.com/archives/C02F27TKQ/p1678721462408249

Upon the new gem version release, some of the notes are not flushed, or the soft flush takes a lot of time.

Expected Behavior

The cache should be purged faster or we should use hard purge for some endpoints where the traffic is not expected to be high but the data consistency may be critical to API users.

Current Behavior

This is visible if you hammer the versions API endpoint (https://rubygems.org/api/v1/versions/karafka.json) a lot. Once in a while (for me that is on average 1/100) the API response does not contain the most recent version

Age on invalid (outdated) requests is always significantly higher. For example (those are response headers):

{"connection"=>"keep-alive", "content-length"=>"71256", "content-type"=>"application/json; charset=utf-8", "x-frame-options"=>"SAMEORIGIN", "x-xss-protection"=>"0", "x-content-type-options"=>"nosniff", "x-download-options"=>"noopen", "x-permitted-cross-domain-policies"=>"none", "referrer-policy"=>"strict-origin-when-cross-origin", "last-modified"=>"Sat, 04 Mar 2023 11:19:17 GMT", "cache-control"=>"max-age=60, public", "content-security-policy"=>"default-src 'self'; font-src 'self' https://fonts.gstatic.com/; img-src 'self' https://secure.gaug.es/ https://gravatar.com/ https://www.gravatar.com/ https://secure.gravatar.com/ https://*.fastly-insights.com https://avatars.githubusercontent.com/; object-src 'none'; script-src 'self' https://secure.gaug.es/ https://www.fastly-insights.com/ 'nonce-'; style-src 'self' https://fonts.googleapis.com/; connect-src 'self' https://s3-us-west-2.amazonaws.com/rubygems-dumps/ https://*.fastly-insights.com https://fastly-insights.com/ https://api.github.com/; form-action 'self' https://github.com/login/oauth/authorize; frame-ancestors 'self'; report-uri https://csp-report.browser-intake-datadoghq.com/api/v2/logs?dd-api-key=pub852fa3e2312391fafa5640b60784e660&dd-evp-origin=content-security-policy&ddsource=csp-report&ddtags=service%3Arubygems.org%2Cversion%3A3e7556c3cbcc6d477eedc5d4ac39ea8bfea451af%2Cenv%3Aproduction%2Ctrace_id%3A3588761172262393171", "x-request-id"=>"6590f3bb-b4d5-4dc4-8939-435338a8a333", "x-runtime"=>"0.053486", "strict-transport-security"=>"max-age=31536000", "x-backend"=>"F_Rails 52.26.151.153:443", "accept-ranges"=>"bytes", "date"=>"Mon, 13 Mar 2023 16:16:06 GMT", "via"=>"1.1 varnish", "age"=>"2371", "x-served-by"=>"cache-ewr18175-EWR", "x-cache"=>"HIT", "x-cache-hits"=>"2", "x-timer"=>"S1678724166.152237,VS0,VE0", "vary"=>"Accept-Encoding", "etag"=>"\"0e03fbb2a341560831aaccdd52e16341\"", "server"=>"RubyGems.org"}

Possible Solution

• Verification that cache invalidation works as expected
• Hard invalidation for data critical endpoints

[simi]

@mensfeld https://github.com/rubygems/rubygems.org/pull/3593 was deployed, feel free to check now.

[simi]

@mensfeld and also small additional fix was deployed later https://github.com/rubygems/rubygems.org/pull/3619

Can you confirm your problem is gone?

[mensfeld]

@simi yes. After the fix and re-sync with RubyGems I do not see this happening anymore.