IAMAuthenticator returned a token that expired in 4 seconds

[leo0514]

We are using IBM VPC go sdk to create VSIs for our service. And we use IamAuthenticator in this go-sdk-core to do the authentication.

       authenticator := &core.IamAuthenticator{
        ApiKey: iamAPIKey,
        URL:    iamTokenServerEndpoint,
    }

    // Create the service options struct.
    options := &vpcv1.VpcV1Options{
        Authenticator: authenticator,
        URL:           vpcURL,
        Version:     VPC_API_VERSION,
    }

        return vpcv1.NewVpcV1(options)

We have been seeing not_authorized errors when creating VSI. We have raised support case for this error and we were told that token by the time reach to storage had expired. In this case, IamAuthenticator returned a token that expired in 4s.

the token expires on
18:24:24Z  and it was received at
18:24:20.434Z"
(4 seconds remaining)

The IBM support told us that the global catalog considers a token expired if it has less than 5 seconds left until expiration when the request is received.

func (authenticator *IamAuthenticator) GetToken() (string, error) {
    if authenticator.getTokenData() == nil || !authenticator.getTokenData().isTokenValid() {
        // synchronously request the token
        err := authenticator.synchronizedRequestToken()
        if err != nil {
            return "", err
        }
    } else if authenticator.getTokenData().needsRefresh() {
        // If refresh needed, kick off a go routine in the background to get a new token
        //nolint: errcheck
        go authenticator.invokeRequestTokenData()
    }

    // return an error if the access token is not valid or was not fetched
    if authenticator.getTokenData() == nil || authenticator.getTokenData().AccessToken == "" {
        return "", fmt.Errorf("Error while trying to get access token")
    }

    return authenticator.getTokenData().AccessToken, nil
}

After checking the code, it seems that the cached token will be used as long as it is not expired even if it will expire within 5 seconds(although it will be refreshed).

It it possible to enhance the code to make sure the returned token's TTL is greater than 15 minutes (or a specified value)?

[padamstx]

Checking with the global catalog service team to confirm server's behavior of (TTL <= 5 secs) --> "expired"

Edit: the global catalog confirmed that the global catalog service does in fact reject a request if the remaining time-to-live for the IAM access token included with the request is <= 5 seconds. This is to avoid the scenario where an access token expires in the middle of a longer-running transaction that might involve other services. In those situations, the user's IAM access token would be propagated to these other services and used for authentication.

[padamstx]

@leo0514 Is this issue relatively easy to reproduce, or did you get unlucky and just happened to invoke an operation within 5 seconds of the access token's time-to-live?

[leo0514]

@leo0514 Is this issue relatively easy to reproduc, or did you get unlucky and just happened to invoke an operation within 5 seconds of the access token's time-to-live?

@padamstx This issue is not easy to reproduce. But we have been seeing this issue lately (few came in last 3 weeks).

[padamstx]

FYI, I've opened PRs to fix this issue in the Go and Java core libraries. The fix is pretty simple... we'll just consider an access token as "expired" when it gets to within 10 seconds of it's IAM-server reported expiration time. I'm working on the Python and Node core libraries now. We should be able to deliver the fixes in all four core libraries within another day or so.

[padamstx]

PR: https://github.com/IBM/go-sdk-core/pull/209

[padamstx]

This issue was fixed in version 5.15.3.