Updating a value doesn't invalidate cache in distributed system

[matteeyah]

Summary

Acutal

If I have two application nodes, one is the web worker, and the other one is a background worker, the following happens:

1. The background worker caches settings upon request
2. The web worker updates a setting
3. This doesn't invalidate the cache in the background worker
4. When executing the background worker it still has the old version of the settings cached and uses that

Maybe sidekiq doesn't process every single job as a new "request" and does some sort of "pooling" or long request 🤔

Expected

1. The background worker caches settings upon request
2. The web worker updates a setting
3. This invalidates the cache in the background worker
4. When executing the background worker it doesn't have the settings cached so it fetches the settings and caches the new value

Workaround

Clear cache before performing each background job with

before_perform { |_| Settings.clear_cache }

System

I'm using ActiveJob with Sidekiq as a backend.

rails (7.0.4)
rails-settings-cached (2.8.3)
sidekiq (7.0.2)

[huacnlee]

rails-settings-cached has solved this problem:

https://github.com/huacnlee/rails-settings-cached/commit/0849bb203e46ba58663ff55288e2589ab501f210

The in-memory cache only enables in Web application process, because by this commit changes, is enabled by a Rails middleware, that now enable in background worker (Sidekiq).

You have not given me the details, which cache storage did you use for? (Redis?)

[huacnlee]

For example:

• A puma process for Web
• 2 Sidekiq workers in background.

Storage cache in Redis.

---

1. When we change settings in Web application, it will update the database and remove the Redis cache in immediately.
2. Next request from Web will fetch settings from Redis cache (if not exist, fallback to database).
3. Next first request from Worker A will fetch Redis cache (if not exist, fallback to database).
4. Next second request from Worker B will get settings directly from Redis cache (because it wrote by Worker A)

[matteeyah]

You have not given me the details, which cache storage did you use for? (Redis?)

The environment in which we were testing wasn't using Redis as the cache store. It used a file store (which was obviously different for every node). Switching to Redis solved this.

Thanks for the help!