[BUG] Authorization requests sometimes returning 401 when running recipes

[davidallendj]

Describe the bug When running the deployment recipes, both SMD and BSS will try to fetch a JWKS from the authorization server (Hydra) to verify incoming JWTs are valid. Hydra will generate a new key pair when the request is made if the pair does not already exist. If both micro-services try to fetch the JWKS roughly at the same time, Hydra will try to generate the pair twice. This will cause all authorization request to return a 401 from both micro-services.

To Reproduce Steps to reproduce the behavior:

1. Make sure the *_JWKS environment variables are set to make SMD and BSS fetch a JWKS
2. Run the deployment recipes
3. Check the Hydra logs to see these lines twice:

   2024-03-05 09:37:13 time=2024-03-05T16:37:13Z level=warning msg=JSON Web Key Set "hydra.jwt.access-token" does not exist yet, generating new key pair... audience=application service_name=Ory Hydra service_version=v2.2.0-rc.3
   2024-03-05 09:37:13 time=2024-03-05T16:37:13Z level=warning msg=JSON Web Key Set "hydra.openid.id-token" does not exist yet, generating new key pair... audience=application service_name=Ory Hydra service_version=v2.2.0-rc.3

4. Fetch a token and try to use with BSS or SMD
5. Returns specific error token is unauthorized

Expected behavior Any normal output expected from the micro-service that isn't token is unauthorized

Desktop (please complete the following information):

• OS: M2 MacOS 14.2 Sonoma

Additional context This problem only occurs sometimes, so you will have to run multiple times if it doesn't happen the first time.

[synackd]

Right now, this is mitigated by introducing a hydra-gen-jwks container that runs curl to cause the JWKS to be generated once. Containers that depend on the JWKS being generated (BSS and SMD), depend on hydra-gen-jwks completing successfully before starting and the race condition is thus avoided. Introduced in #25.

Ideally, hydra would address this but this works until they do.

With the above, I think we can close this since tests pass both using Docker on the test cluster and Docker Desktop.