Improve Cody Gateway reliability when fetching subscription data for rate limits

[vdavid]

Problems

• Refreshing plan data happens every 15 min, after the cache is expired. It is currently a blocking action. It increases the latency of Cody endpoints when this happens.
• If dotcom is unavailable when we check, we return 403 and we block Cody usage. And dotcom is often unavailable.

Solution ideas

• Refresh plan data before it expires
• Increase timeout

[vdavid]

We're also getting a dozen messages a day on #alerts-cody-gateway (from this line of code, example Sentry item), which, I think, the result of this issue.

[vdavid]

@thenamankumar, do you want to tackle this this week? I'm fine with doing it later, but I remember that last week you were pretty positive about finishing this soon. Please update its status according to your answer

[thenamankumar]

No plans to take this up for this week yet.

[chrsmith]

Moving this out of February. For the current release, Cody Gateway will continue to fetch subscription information from dotcom (and NOT SSC directly).

There are howeve plans for some other situations where SSC may contact Cody Gateway directly (e.g. forcing a refresh of a user's rate limits in response to them changing their subscription plan.)

As far as Cody Gateway handling temporary errors when looking up subscription information from dotcom, that would be considered a bug fix but does not meet the bar for February.

Update (3/13): Now that the February release is complete, we can/should update Cody Gateway to fetch the subscription information directly from the SSC backend. (This simplifies the number of moving parts, and other improvements.)

However, realistically we first need to address https://github.com/sourcegraph/accounts.sourcegraph.com/issues/642 before we can get started on this. (Since otherwise we'd be in an even more awkward state while we transition things.)