Api call not retried on token expiration (PrivateKey mode)

[gao-artur]

This is the same issue like Intermittent 401s from Okta SDK with Private Key AuthMode (Token Expired). The difference is 401 are not intermittent, they happen once in a hour that aligns with token lifetime. Looking into logs it seems the sdk refreshes the token after receiving 401 but not retries the api call. Here are trace logs with timestamps:

2022-01-26 08:06:38,631 [TRACE] GET api/v1/logs?filter=....
2022-01-26 08:06:38,709 [TRACE] 401 /api/v1/logs?filter=...
2022-01-26 08:06:38,709 [TRACE] Generate a signed JWT
2022-01-26 08:06:38,711 [TRACE] Request an access token.
2022-01-26 08:06:39,268 [ERROR] Failed to get logs from Okta
Okta.Sdk.OktaApiException:  (401, )
   at Okta.Sdk.Internal.DefaultDataStore.EnsureResponseSuccess(HttpResponse`1 response)
   at Okta.Sdk.Internal.DefaultDataStore.GetArrayAsync[T](HttpRequest request, RequestContext requestContext, CancellationToken cancellationToken)
   at Okta.Sdk.PagedCollectionEnumerator`1.MoveNextAsync()
   at Okta.Sdk.Internal.CollectionAsyncEnumerator`1.MoveNextAsync()
   at System.Linq.AsyncEnumerablePartition`1.SkipAndCountAsync(UInt32 index, IAsyncEnumerator`1 en) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 377
   at System.Linq.AsyncEnumerablePartition`1.SkipAndCountAsync(Int32 index, IAsyncEnumerator`1 en) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 371
   at System.Linq.AsyncEnumerablePartition`1.SkipBeforeAsync(Int32 index, IAsyncEnumerator`1 en) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 365
   at System.Linq.AsyncEnumerablePartition`1.TryGetFirstAsync(CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 261
   at System.Linq.AsyncEnumerablePartition`1.TryGetFirstAsync(CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerablePartition.cs:line 261
   at System.Linq.AsyncEnumerable.<FirstOrDefaultAsync>g__Core|287_0[TSource](IAsyncEnumerable`1 source, CancellationToken cancellationToken) in /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/FirstOrDefault.cs:line 33

I'm using Okta.Sdk v5.3.1.

[andriizhegurov-okta]

Hi @gao-artur, I apologize for the inconvenience, we already have a ticket in our backlog to address this issue.

Just to clarify, can you confirm that you are requesting a new token with your custom implementation of the IOAuthTokenProvider interface as it's said in the Okta .NET management SDK Readme?

[gao-artur]

Hey @andriizhegurov-okta, no, I'm using default implementation. I actually see that there is a retry for 401 when working in PrivateKey mode. I tried to debug the code, the new access token generated and applied on Authorization header, but okta returns 401 also on retry.

[gao-artur]

Hey @andriizhegurov-okta, I found the bug.

// If OAuth token expired, get a new token and retry
if ((int)response.StatusCode == 401 &&
    (_oktaConfiguration.AuthorizationMode == AuthorizationMode.PrivateKey ||
     _oktaConfiguration.AuthorizationMode == AuthorizationMode.BearerToken))
{
    await ApplyOAuthHeaderAsync(true).ConfigureAwait(false);

    // Sending same request twice cause failures
    var clonedRequest = await HttpRequestMessageHelper.CloneHttpRequestMessageAsync(request).ConfigureAwait(false);

    using (response = await _retryStrategy.WaitAndRetryAsync(clonedRequest, cancellationToken, operation).ConfigureAwait(false))
    {
        return await ProcessResponseAsync(response).ConfigureAwait(false);
    }
}

The retry code applies new token on _httpClient.DefaultRequestHeaders.Authorization header. Then HttpRequestMessageHelper.CloneHttpRequestMessageAsync copies headers from old request object

foreach (KeyValuePair<string, IEnumerable<string>> header in request.Headers)
{
    clonedRequest.Headers.TryAddWithoutValidation(header.Key, header.Value);
}

One of them is Authorization header with old expired bearer token. This header overrides _httpClient.DefaultRequestHeaders.Authorization. Excluding Authorization header from copy solves the issue.

[andriizhegurov-okta]

Hi @gao-artur, thanks for the investigation. As I said we have a ticket to fix this issue. I've raised the priority of the ticket and we'll have that fixed as soon as possible.

BTW, have you already tried this solution?

[gao-artur]

Yes, it worked

[gao-artur]

@andriizhegurov-okta any ETA fixing this bug? It blocks us from replacing api token with private key.

[andriizhegurov-okta]

@gao-artur We plan to release the fix in the next version of the Okta.Sdk. Unfortunately I can't provide an exact ETA, but most likely this new version will be released next week.

[andriizhegurov-okta]

@gao-artur this should be fixed in Okta.Sdk v5.4.1

[gao-artur]

Thank you! Will verify next week.

[gao-artur]

@andriizhegurov-okta it seems to work now. You can close the issue, thanks!