How does config rotation work?

[ehaydenr]

I'm thinking of the following scenario:

• t=0 OHTTP Gateway gives config A to mirror with validity of 10 mins
• t=5min OHTTP Gateway wants to rotate config to config B
• t=6min client fetches config from OHTTP gateway, gets config B
• t=6min client goes to mirror and is provided config A because it's still valid in cache. Inconsistency

If the client is checking multiple mirrors, a similar inconsistency could happen since mirrors will be refreshing their cache at different times.

[chris-wood]

This seems like a misconfiguration for the gateway since they specified a validity window that was longer than that which was actually used in practice. I think the expectation is that the cache validity must always be less than the lifetime of the resource.

[ehaydenr]

What would a valid rotation timeline look like?

If I'm an OHTTP gateway that normally advertises a 10 minute validity window, do I have to disable caching of my configuration altogether for at least 10 minutes so no mirrors cache while I rotate?

Even if I did that, I think there's still this race condition:

• right before gateway config rotation, client fetches from gateway (tiny/no validity window)
• gateway rotates
• client goes to mirror, gets the new config.. inconsistency

[chris-wood]

@ehaydenr by gateway are you referring to an OHTTP gateway, or the origin which provides the source of truth for a resource?

In any case, I think you're right that this race condition will exist even if there were no cache controls on the resource. Probably the simplest way to deal with this is for clients to retry their request to the gateway after the mirror to see if the resource has changed. Since the error path here depends on client policy, i.e., does it treat mismatches as failures and abort, does it retry the whole flow, etc, I think it's reasonable to treat this as an edge case that implementations work around.

@tfpauly, thoughts?