Support Forest Client: BC Service Card login issue

[MCatherine1994]

Describe the task Team Alliance is getting bc service card login issue. They found that when try to login using test BC Service Card account, sometimes they will get a read time out error. I found that as well sometimes when using the OIDC debugger tool. And if try to login again, it works fine, the login will be successful.

The original email from Maria is in the screenshot:

Acceptance Criteria

• [x] Find out what causes the issue

Additional context

• Send the email to Wesley, but starts to feel the timeout might happen on our side.
• From Ian: Based on the problem description (if I understand correctly), you wont be finding Cloud Watch log for our lambda log (our FAM log) but might be the log that AWS auto created for lambda (access log). Because what I think is if the assumption or first time reaching the lambda when it is COLD is true, AWS is invocating the lambda (thus not yet executing the handler within our FAM code yet) and our Cognito receive time-out and thus generating 302 redirect for the user browser. If you look at the sequence diagram, when the “code” received from the BCSC, browser then hand over to Cognito, which then tries to pass the “code” to our lambda. But because our lambda is COLD so time-out for Cognito. Then the rest of the happy path sequence diagram isn’t fulfilled.
• From Basil: We should be looking at either cognito timeout settings (to make the timeout duration longer), or look at the lambda configuration to see if we can keep it hotter. Also, it might not be the lambda itself – could be the database proxy

[MCatherine1994]

API Lambda logs when bc service card login fail, it generates 2 logs:

1st one: log-events-viewer-result.csv

2st one: log-events-viewer-result (1).csv

Try to login again, and successfully login, it generates 1 more log, and add logs in the last 2:

The last 2 logs with more information: log-events-viewer-result (2).csv log-events-viewer-result (3).csv

The new generated log: log-events-viewer-result (4).csv

[MCatherine1994]

From the AWS documentation https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools-working-with-aws-lambda-triggers.html, when Amazon Cognito calls your Lambda function, it must respond within 5 seconds.

When checking the logs in the login failure case, the call to get the bc service card token takes about 3s, in the login successful case, it takes about 0.5s.

code where we make the call:

failure case:

successful case:

So next step we need to contact Wesley and ask about the response time for making the call to https://idtest.gov.bc.ca/oauth2/token

[basilv]

Decided resolution is for the BCSC team to cache (hardcode) our public key in their system, to avoid the ~2 sec key lookup overhead that was causing the Cognito get user info call to our get user info endpoint to timeout after 5 seconds.

I've researched our use of the key and documented it in our secret registry that when we change it we'll need to update the BCSC team.