[Bug] OBO token is not refreshed when it has expired

[jmprieur]

Logs and Network traces Without logs or traces, it is unlikely that the team can investigate your issue. Capturing logs and network traces is described at https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/logging

Which Version of MSAL are you using ? MSAL.NET 4.29

Platform .NET Core

What authentication flow has the issue?

• Web API
  • [ x ] OBO

Repro

See https://github.com/AzureAD/microsoft-identity-web/compare/jmprieur/repro_obo_rt. This is simulating a long running process in a web API, or a feature like OneDrive that creates albums for the user who is no longer signed-in. This requires the OBO token to be refreshed, (which is possible until the refresh token expires).

A controller action calls RegisterPeriodicCallbackForLongProcessing() which itself registers a callback (with a timer)

        /// <summary>
        /// This methods the processing of user data where the web API periodically checks the user
        /// date (think of OneDrive producing albums)
        /// </summary>
        private void RegisterPeriodicCallbackForLongProcessing()
        {
            // Get the token incoming to the web API - we could do better here.
            var token = (HttpContext.Items["JwtSecurityTokenUsedToCallWebAPI"] as JwtSecurityToken).RawData;

            // Build the URL to the callback controller, based on the request.
            var request = HttpContext.Request;
            string url = request.Scheme + "://" + request.Host + request.Path.Value.Replace("todolist", "callback");

            // Setup a timer so that the API calls back the callback every 10 mins.
            Timer timer = new Timer(async (state) =>
            {
                HttpClient httpClient = new HttpClient();
                httpClient.DefaultRequestHeaders.Add("Authorization", $"bearer {token}");
                var message = await httpClient.GetAsync(url);
            }, null, 1000, 1000 * 60 * 10);
        }

The callback is defined in another controller CallbackController

       [HttpGet]
        public async Task GetAsync()
        {
            _logger.LogWarning($"Callback called {DateTime.Now}");
            string owner = User.GetDisplayName();
            // Below is for testing multi-tenants
            var result = await _tokenAcquisition.GetAuthenticationResultForUserAsync(new string[] { "user.read" }).ConfigureAwait(false); // for testing OBO

            if (result.AuthenticationResultMetadata.TokenSource == Microsoft.Identity.Client.TokenSource.IdentityProvider)
            {
              _logger.LogWarning($"OBO access token refreshed {DateTime.Now}");
            }
        }

Expected behavior Passing the same (expired) incoming token (as a key), given that there is a refresh token associated to the (expired) OBO access token that was generated from the incoming (expired) token, this OBO token should be refreshed, and the long running process should happen for ever (as the refresh token will also be refreshed). See the protocol documentation for On behalf of. The refresh token is returned (when offline_access is requested), for this scenario.

Therefore , with the repro code above, we should see, after 1h, a log warning "OBO access token refreshed "

Actual behavior EVO does not accept the incoming (expired) token, of course, but MSAL.NET doesn't even attempt to refresh the OBO access token, and there is an exception.

Microsoft.Identity.Client.MsalUiRequiredException HResult=0x80131500 Message=AADSTS500133: Assertion is not within its valid time range. Ensure that the access token is not expired before using it for user assertion, or request a new token. Current time: 2021-04-17T19:50:21.6663073Z, expiry time of assertion 2021-04-17T19:50:15.0000000Z. Trace ID: ee96f1de-d755-46e7-811e-aba122bd4700 Correlation ID: 1a64be83-e614-444a-a7ce-7c2f570cf35a Timestamp: 2021-04-17 19:50:21Z Source=Microsoft.Identity.Client StackTrace: at Microsoft.Identity.Client.Internal.Requests.RequestBase.HandleTokenRefreshError(MsalServiceException e, MsalAccessTokenCacheItem cachedAccessTokenItem) at Microsoft.Identity.Client.Internal.Requests.OnBehalfOfRequest.d2.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Microsoft.Identity.Client.Internal.Requests.RequestBase.d13.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.<ExecuteAsync>d__4.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.Identity.Web.TokenAcquisition.d__26.MoveNext() in C:\gh\microsoft-identity-web\src\Microsoft.Identity.Web\TokenAcquisition.cs:line 631

This exception was originally thrown at this call stack: Microsoft.Identity.Client.Internal.Requests.RequestBase.HandleTokenRefreshError(Microsoft.Identity.Client.MsalServiceException, Microsoft.Identity.Client.Cache.Items.MsalAccessTokenCacheItem) Microsoft.Identity.Client.Internal.Requests.OnBehalfOfRequest.ExecuteAsync(System.Threading.CancellationToken) System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult() Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(System.Threading.CancellationToken) System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task) System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) ... [Call Stack Truncated]

[jmprieur]

Note that to repro this, if you don't change the token lifetime (delay for expiry), you'll have to be patient :)

[bgavrilMS]

refres_token grant asks for a new AT based on:

• RT
• scopes
• tenant

We need clarification if this brings back an AT that is always valid for OBO

[SirElTomato]

@jmprieur is there any progress with this issue?

It's blocking us adding a feature to our app.

Here is a reminder of what it does The user signs in from the mobile app and gives permission to read emails. From our webapi, we use OBO to cache the tokens so that we can perform a long running scan of the users inbox, and repeat these scans at any point the user wants.

[jmprieur]

@bgavrilMS : I'd like prioritize fixing this

[bgavrilMS]

I'm working on documenting the changes needed to OBO and can work on a prototype for this.

[bgavrilMS]

Support for this has been completed in MSAL, see https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/issues/2623

Microsoft.Identity.Web may choose to further improve this scenario in ASP.NET Core, as it requires hanging on to the original assertion, even if it has expired.

[SirElTomato]

Thanks for the fix.

@jmprieur I am now getting the following error after updating the Microsoft.Identity.Client package to 4.33.0:

Code: authenticationChallengeRequired
Message: Authentication challenge is required.

I am creating and using the graph client like this:

var authProvider = new AuthorizationCodeProvider(_revokeMicrosoftService.ConfidentialClientApplication, _scopes);
_graphClient = new GraphServiceClient(authProvider);
var mailFolders = await _graphClient.Me.MailFolders.Request().GetAsync();

I have tried changing it to use ClientCredentialProvider instead of AuthorisationCodeProvider (can you confirm if this is correct?) but now get the following error:

AADSTS50059: No tenant-identifying information found in either the request or implied by any provided credentials.
Trace ID: eaef4a24-4848-4dd2-9d61-4c33be89e900
Correlation ID: 397c2d4d-b77e-4767-9a78-f659118296bb
Timestamp: 2021-06-29 10:43:59Z

var authProvider = new ClientCredentialProvider(_revokeMicrosoftService.ConfidentialClientApplication, "api://<applicationId>/.default");
_graphClient = new GraphServiceClient(authProvider);
var mailFolders = await _graphClient.Me.MailFolders.Request().GetAsync();

Please can you advise what I am doing wrong?

Thanks

[jmprieur]

@SirElTomato : this is not OBO: OBO is for web APIs calling downstream web APIs. AuthorizationCodeProvider is for web apps that calls APIs ClientCredentialProvider for apps (web apps, web APIs, console apps), that call APIs on their own behalf: that is daemon apps

• What are you building? a web app? or a web API?
• Which technology? ASP.NET Core? or ASP.NET classic ?

[SirElTomato]

@jmprieur

• What are you building? a web app? or a web API?
  • The user signs in from the mobile app and gives permission to read emails. From our webapi, we use OBO to cache the tokens so that we can perform a long running scan of the users inbox, and repeat these scans at any point the user wants.
• Which technology? ASP.NET Core? or ASP.NET classic ?
  • ASP.NET classic

If it is neither AuthorizationCodeProvider or ClientCredentialProvider, what should I be using instead?

[jmprieur]

@SirElTomato then I think you want to use the On-behalf-of provider: https://docs.microsoft.com/en-us/graph/sdks/choose-authentication-providers?tabs=CS#OnBehalfOfProvider

[SirElTomato]

@jmprieur Perfect that worked! Thank you for all your help

[SirElTomato]

@jmprieur I am still getting the following error after an hour. I have updated Microsoft.Identity.Client to 4.33.0

{"AADSTS70000: The provided value for the 'assertion' is not valid. The assertion has expired.\r\nTrace ID: c7d444bc-e1f6-4304-ab54-dbd3c7500c00\r\nCorrelation ID: 20d3aea2-01c3-4105-a81d-5e31ab8a77f5\r\nTimestamp: 2021-07-05 13:16:43Z"}

[SirElTomato]

Here is a breakdown of the flow I am using:

• Login from mobile app to retrieve JWT.
• Send JWT to our API and execute the following to acquire a token and cache the token in cosmos:

  var userAssertion = new UserAssertion(jwt);
  var res = await ConfidentialClientApplication.AcquireTokenOnBehalfOf(scopes, userAssertion).ExecuteAsync();

• Save the JWT to the user object so that it can be used to acquire the token again in future
• Call this again when a token is needed to call the Outlook api using the original jwt which was saved to the user object

  var userAssertion = new UserAssertion(jwt);
  var res = await ConfidentialClientApplication.AcquireTokenOnBehalfOf(scopes, userAssertion).ExecuteAsync();

Other related code:

private IConfidentialClientApplication BuildApp()
        {
            // The application which retreives the auth token must also use the same client id and redirect url when requesting the token. The auth token from the response is then passed to the AquireTokenByAuthorisationCode method below. 

            IConfidentialClientApplication app = ConfidentialClientApplicationBuilder
                .Create(_authenticationSettings.MicrosoftRevokeAppId)
                .WithRedirectUri(_authenticationSettings.EmailScanWebAppBaseUrl + _authenticationSettings.MicrosoftRedirectAction)
                .WithClientSecret(_authenticationSettings.MicrosoftRevokeAppSecret)
                .WithTenantId(_authenticationSettings.MicrosoftTenantId)
                .Build();

            IMsalTokenCacheProvider cosmosTokenCacheProvider = CreateCosmosTokenCacheSerialiser();
            Task.Run(() => cosmosTokenCacheProvider.InitializeAsync(app.UserTokenCache)).Wait();

            return app;
        }

        private IMsalTokenCacheProvider CreateCosmosTokenCacheSerialiser()
        {
            IServiceCollection services = new ServiceCollection();
            var cosmosConnectionString = string.Format("AccountEndpoint={0};AccountKey={1};", _databaseSettings.uri, _databaseSettings.key);

            services.AddDistributedTokenCaches();
            services.AddCosmosCache((CosmosCacheOptions cacheOptions) =>
            {
                cacheOptions.DatabaseName = _databaseSettings.db;
                cacheOptions.ContainerName = _authenticationSettings.CosmosMsalContainer;
                cacheOptions.CreateIfNotExists = true;
                cacheOptions.ClientBuilder = new CosmosClientBuilder(cosmosConnectionString);
            });

            IServiceProvider serviceProvider = services.BuildServiceProvider();
            IMsalTokenCacheProvider msalTokenCacheProvider = serviceProvider.GetRequiredService<IMsalTokenCacheProvider>();

            return msalTokenCacheProvider;
        }

This works fine until an hour has elapsed since the original authentication then it starts failing with this error:

{"AADSTS70000: The provided value for the 'assertion' is not valid. The assertion has expired.\r\nTrace ID: c7d444bc-e1f6-4304-ab54-dbd3c7500c00\r\nCorrelation ID: 20d3aea2-01c3-4105-a81d-5e31ab8a77f5\r\nTimestamp: 2021-07-05 13:16:43Z"}

[SirElTomato]

@trwalke I don't believe this is fixed, please see my comments and suggest where I am going wrong if that is the case

[SirElTomato]

@bgavrilMS @jmprieur This is still broken, still getting this.

MSAL.Desktop.4.35.0.0.MsalUiRequiredException: ErrorCode: invalid_grant Microsoft.Identity.Client.MsalUiRequiredException: AADSTS70000: The provided value for the 'assertion' is not valid. The assertion has expired. Trace ID: cc8a7860-b86c-4394-a7bd-33395300c300 Correlation ID: f93e1e0c-01b0-4f15-9ac6-85017235b26f Timestamp: 2021-08-02 10:46:32Z at Microsoft.Identity.Client.Internal.Requests.RequestBase.<HandleTokenRefreshErrorAsync>d__24.MoveNext()

[bgavrilMS]

Hi @SirElTomato - can you provide some logs please? I'd like to better understand what is happening. https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/logging

The idea here is that a refresh token is cached. If the existing AT (obtained via OBO) has expired, then the RT is refreshed (i.e. refresh_token flow) instead of the OBO flow. We chose this pattern because the OBO flow fails with this error when the original assertion expires.

[SirElTomato]

@bgavrilMS are you saying that I need to change my code to use the refresh token flow or this is how it works in the background?

I will get some logs for you. Where can I send the logs to, assuming you want pii info included?

[bgavrilMS]

No need to change your code, this should work, we need to investigate what is happening.

[SirElTomato]

@bgavrilMS where can I send the logs to?

[SirElTomato]

I have sent them to your Microsoft email

[bgavrilMS]

No access tokens found in the cache. Skipping filtering.

I had a look at the logs and it seems that the OBO call is not able to find the correct token cache entry. So OBO behaves as if there are no tokens cached - it takes the original assertion and tries to obtain a new token from AAD. But since the original assertion has already expired, you get this exception. What we'd want to happen however is for OBO to find a pair of (AT, RT) in the cache, where the AT will also have expired. OBO will then automatically use the RT to get a new AT.

Thoughts on how to ensure that the correct token cache is loaded? It's not clear to me how you use Microsoft.Identity.Web IMsalTokenCacheProvider. Since you are creating your own ConfidentialClientApplication object, I assume you only rely on Microsof.Identity.Web for the cache adaptors?

In this case, did you forget to call:

msalTokenCacheProvider.Initialize(app.UserTokenCache)

[SirElTomato]

@bgavrilMS you can see the code I am using above or here

I am calling

IMsalTokenCacheProvider cosmosTokenCacheProvider = CreateCosmosTokenCacheSerialiser();
cosmosTokenCacheProvider.Initialize(app.UserTokenCache);

Tokens are stored in a Cosmos container using the Microsoft Cosmos Caching package

[bgavrilMS]

I'm not very familiar with ASP.NET Core's DI model to fully understand if you are injecting the token cache or not. Can you see in Cosmos if values are written or not? I expect the cache key to be Hash(original_assertion) so a fairly random string.

Also I see that InitializeAsync from Task.Run(() => cosmosTokenCacheProvider.InitializeAsync(app.UserTokenCache)).Wait(); was deprecated, can you confirm that you use Initialize ?

@jennyf19 - could you take a look at how the token cache is defined here? How can we get some logs out of Microsoft.Identity.Web just for the cache?

[jennyf19]

@SirElTomato thanks for all the details, i'll need some time to catch up on this. We have simplified the cache serialization, if you are interested: example here.

If you could enable logging, you would need to install Microsot.Extensions.Logging, and set the level to verbose, we can get specific logs going to the distributed cache. Will try to get to this today. thanks.

[SirElTomato]

@bgavrilMS yes the cache is populated with data after the initial log in

[SirElTomato]

@jennyf19 do you need more logs other than the ones I have given to @bgavrilMS ?

[jennyf19]

@SirElTomato I haven't seen the logs yet, didn't know you had already sent something. Will let you know. thanks.

[bgavrilMS]

I'll share the MSAL logs with @jennyf19. The MSAL logs simply show that now AT or RT were found in MSAL's memory. This can be due to:

1. this is the first time you call OBO with that assertion OR
2. something is wrong with the cache and you do not locate the right cache node to feed it to MSAL

I think that next we need to understand why there are no cached tokens.

@SirElTomato - could it be due to 1? You have to call OBO once during the lifetime (1h) of the assertion from the mobile app, otherwise AAD will not respond to OBO. But once you do that, AAD will give you back an AT and a refresh token (RT), which MSAL can use.

[SirElTomato]

@bgavrilMS I am calling OBO with the assertion immediately after logging in from the mobile app as described above (or here. When this is called, an entry is added to the MSAL cache Cosmos container. I can then successfully get a token for calling a downstream api (outlook graph api) within an hour. After an hour has elapsed, I can no longer get a token.

[SirElTomato]

Mobile app code:

    private readonly kScopes = [
        'api://6480c91c-8732-4289-91f2-76c7635bb240/ScanEmails'
    ];

    private readonly kExtraScopes = [
        "mail.read"
    ];
    private readonly kClientId = '6480c91c-8732-4289-91f2-76c7635bb240';

    private _msalClient: PublicClientApplication;

    constructor() {
        this._msalClient = new PublicClientApplication({ auth: { clientId: this.kClientId } });
    }

    login() {
        return new Promise<boolean>(async (resolve) => {

            const acquireTokenParams: MSALInteractiveParams = {
                scopes: this.kScopes,
                promptType: MSALPromptType.CONSENT,
                extraScopesToConsent: this.kExtraScopes
            };

            try {
                const result: MSALResult = await this._msalClient.acquireToken(acquireTokenParams);
                const resultString = JSON.stringify(result);
                if (result && result.accessToken) {
                    console.log(result.accessToken);
                    const response = await myApi.GenerateMicrosoftApiTokens({ jwt: result.accessToken, scopes: this.kScopes })
                    resolve(response.success);
                }
                else {
                    resolve(false);
                }
                console.log(resultString);

            } catch (error) {
                const jsonError = JSON.stringify(error);
                console.log(jsonError);
                resolve(false);
            }
        })
    }

Service which handles the MSAL auth

public class MicrosoftService : IMicrosoftService
{
    public IConfidentialClientApplication ConfidentialClientApplication { get; set; }
    private readonly ICompanyUserRepo _userRepo;
    private readonly IAuthenticationSettings _authenticationSettings;
    private readonly IDatabaseSettings _databaseSettings;

    public MicrosoftService(
        ICompanyUserRepo userRepo,
        IAuthenticationSettings configuration,
        IDatabaseSettings databaseSettings)
    {
        _userRepo = userRepo;
        _authenticationSettings = configuration;
        _databaseSettings = databaseSettings;

        ConfidentialClientApplication = BuildApp();
    }

    public async Task GetAccessTokenAndUpdateUser(string userId, IEnumerable<string> scopes, string jwt)
    {
        await AquireTokenOnBehalfOf(scopes, jwt);

        var user = _userRepo.GetByIdAndPartitionKeyOrDefault(userId, userId);
        user.MicrosoftIdentity = jwt;

        _userRepo.Update(user);
    }

    public async Task<AuthenticationResult> GetToken(string userId, IEnumerable<string> scopes)
    {
        var user = _userRepo.GetByIdAndPartitionKeyOrDefault(userId, userId);
        var jwt = user.MicrosoftIdentity;

        return await AquireTokenOnBehalfOf(scopes, jwt);
    }

    public async Task RevokeToken(string userId, IEnumerable<string> scopes)
    {
        var user = _userRepo.GetByIdAndPartitionKeyOrDefault(userId, userId);
        var authResult = await AquireTokenOnBehalfOf(scopes, user.MicrosoftIdentity);

        await ConfidentialClientApplication.RemoveAsync(authResult.Account);
        user.MicrosoftIdentity = null;

        _userRepo.Update(user);
    }

    private async Task<AuthenticationResult> AquireTokenOnBehalfOf(IEnumerable<string> scopes, string jwt)
    {
        var userAssertion = new UserAssertion(jwt);
        var res = await ConfidentialClientApplication.AcquireTokenOnBehalfOf(scopes, userAssertion).ExecuteAsync();

        return res;
    }

    private IConfidentialClientApplication BuildApp()
    {
        // The application which retreives the auth token must also use the same client id and redirect url when requesting the token. The auth token from the response is then passed to the AquireTokenByAuthorisationCode method below. 

        IConfidentialClientApplication app = ConfidentialClientApplicationBuilder
            .Create(_authenticationSettings.MicrosoftRevokeAppId)
            .WithRedirectUri(_authenticationSettings.EmailScanWebAppBaseUrl + _authenticationSettings.MicrosoftRedirectAction)
            .WithClientSecret(_authenticationSettings.MicrosoftRevokeAppSecret)
            .WithTenantId(_authenticationSettings.MicrosoftTenantId)
            .Build();

        IMsalTokenCacheProvider cosmosTokenCacheProvider = CreateCosmosTokenCacheSerialiser();
        cosmosTokenCacheProvider.Initialize(app.UserTokenCache);

        return app;
    }

    private IMsalTokenCacheProvider CreateCosmosTokenCacheSerialiser()
    {
        IServiceCollection services = new ServiceCollection().AddLogging();
        var cosmosConnectionString = string.Format("AccountEndpoint={0};AccountKey={1};", _databaseSettings.RevokeGlobalDatabaseUri, _databaseSettings.RevokeGlobalDatabaseKey);

        services.AddDistributedTokenCaches();
        services.AddCosmosCache((CosmosCacheOptions cacheOptions) =>
        {
            cacheOptions.DatabaseName = _databaseSettings.RevokeGlobalDatabase;
            cacheOptions.ContainerName = _authenticationSettings.CosmosMsalContainer;
            cacheOptions.CreateIfNotExists = true;
            cacheOptions.ClientBuilder = new CosmosClientBuilder(cosmosConnectionString);
        });

        IServiceProvider serviceProvider = services.BuildServiceProvider();
        IMsalTokenCacheProvider msalTokenCacheProvider = serviceProvider.GetRequiredService<IMsalTokenCacheProvider>();

        return msalTokenCacheProvider;
    }
}

}

My api service to call downstream api (outlook)

public class OutlookService : IOutlookService
    {
        private readonly List<string> _scopes = new List<string> { "offline_access", "mail.read", "openid" };
        private readonly IMicrosoftService _microsoftService;
        private readonly GraphServiceClient _graphClient;

        public OutlookService(IMicrosoftService microsoftService)
        {
            _microsoftService = microsoftService;

            var authProvider = new OnBehalfOfProvider(_microsoftService.ConfidentialClientApplication, _scopes);
            _graphClient = new GraphServiceClient(authProvider);
        }

        public async Task<EmailPagingContent> GetSenders(string userId, string pagingLink, int numberOfEmailsToGetPerPage, string folderToScanDisplayName)
        {
            var token = await _revokeMicrosoftService.GetToken(userId, _scopes);

            var mailFolders = await _graphClient.Me.MailFolders.Request().GetAsync();
            MailFolder mailFolder = mailFolders.ToList().Find(x => x.DisplayName == folderToScanDisplayName);

            if (mailFolder == null)
            {
                throw new ArgumentException("Folder to scan display name does not exist.");
            }

            var queryOptions = new List<QueryOption>();

            if (!string.IsNullOrEmpty(pagingLink))
            {
                new List<QueryOption> {
                    (new QueryOption("$skiptoken", pagingLink)),
                };
            }

            var messages = await _graphClient.Me.MailFolders[mailFolder.Id].Messages.Request(queryOptions)
                .Top(numberOfEmailsToGetPerPage)
                .OrderBy("receivedDateTime desc")
                .GetAsync();

            var emailContent = messages.Select(x => new EmailContent { Id = x.Id, Sender = x.Sender.EmailAddress.Address }).ToList();
            pagingLink = messages.NextPageRequest?
                .QueryOptions?
                .FirstOrDefault(x => string.Equals("$skiptoken", x.Name, StringComparison.InvariantCultureIgnoreCase))?
                .Value;

            return new EmailPagingContent()
            {
                EmailContent = emailContent,
                PagingLink = pagingLink,
            };
        }
    }

[SirElTomato]

• Login from mobile app to retrieve JWT.
• Send JWT to our API and execute the following to acquire a token and cache the token in cosmos:

  var userAssertion = new UserAssertion(jwt);
  var res = await ConfidentialClientApplication.AcquireTokenOnBehalfOf(scopes, userAssertion).ExecuteAsync();

• Save the JWT to the user object so that it can be used to acquire the token again in future
• Call this again when a token is needed to call the Outlook api using the original jwt which was saved to the user object

var userAssertion = new UserAssertion(jwt);
var res = await ConfidentialClientApplication.AcquireTokenOnBehalfOf(scopes, userAssertion).ExecuteAsync();

[bgavrilMS]

Ok, so the first time the service works, OBO gets tokens and stores them in the cache. Are there any eviction policies on the Cosmos cache? Maybe the entry gets deleted before MSAL needs to read it again?

We'd really need to see what's happening in the cache logic, which you get from Microsoft.Identity.Web. Can you try to add logging as described here: https://github.com/AzureAD/microsoft-identity-web/wiki/Logging ? You should see messages like these https://github.com/AzureAD/microsoft-identity-web/blob/master/src/Microsoft.Identity.Web/TokenCacheProviders/Distributed/MsalDistributedTokenCacheAdapter.Logger.cs#L17

[SirElTomato]

@bgavrilMS are there similar instructions for .net framework (4.7.2)?

[bgavrilMS]

Ah, I do not know. @jennyf19 or @jmprieur can you please advise? I think the token caching is not set up correctly. The caches are from Microsoft.Identity.Web, but the app is .net classic.

[jennyf19]

@SirElTomato have you seen the guidance on implementing a long running process, and this sample.

@bgavrilMS can you send me the logs that you mentioned earlier? thx.

[SirElTomato]

@jennyf19 the problem isn't that it is a long running process. The problem is that trying to get a token fails when an hour has elapsed since the initial authentication and token caching, irrespective of how long anything takes.

[jmprieur]

@SirElTomato. Let's recap what I understand;

• you have a protected web API, for which a client has acquired a token.
• from the web api request a token for a downstream web api using acquireTokenOnBehalfOf.
• after one hour this fails. But what fails?
  • can you still use the token from the client to call your web api ? (you shouldn't).
  • or do have your web api call itself through a long running process, and use that token (which has expired from the purpose of calling your web api, but can still be used as a key to get the OBO Token from the cache and refresh it)

Do you have a kind of schema of your architecture? and more details about what fails where?

[SirElTomato]

@jmprieur this is the what I am doing. Is this not correct? Should I not be using the original jwt when calling acquireTokenOnBehalfOf even though this works within in the first hour?

[SirElTomato]

@jmprieur ?

[jmprieur]

@SirElTomato : you should use the token that was used to call you own web API (not the token that you got from Microsoft.Identity.Web)

[SirElTomato]

@jmprieur I am using the token that I am using to call my own web API as described here

[SirElTomato]

the following from the app code

const response = await myApi.GenerateMicrosoftApiTokens({ jwt: result.accessToken, scopes: this.kScopes })

calls GetAccessTokenAndUpdateUser on the MicrosoftService

[SirElTomato]

@jmprieur how can I change the token lifetime so that this is easier to test?

[bgavrilMS]

@SirElTomato - I believe you can use a conditional access policy to set the AT lifetime to 10 min https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-configurable-token-lifetimes

Why I suspect is happening in this case is a cache miss problem.

• the first time you call OBO, MSAL will cache the response using Hash(mobile_app_access_token)
• the second time you call OBO, MSAL first tries to look into the cache for that key, but for some reason it is not there.

We can confirm this by adding some logging for the token cache itself, have a look at:

https://github.com/AzureAD/microsoft-identity-web/wiki/Logging#logging-in-net-framework-or-net-core

We are working on improving this scenario by the way, to allow app developers to use they own keys which better describe the session. https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/issues/2733

[SirElTomato]

@bgavrilMS I have sent you some logs via email

[bgavrilMS]

Thanks for the latest round of logs @SirElTomato. I had a look and double checked OBO functionality, but can't seem to find the issue.

In your logs you have shown me 3 entries. These are simply base64 encoded over some JSON which is pretty easy to understand.

1st and 2nd entries are tokens associated with assertion hashes. Inside the payloads are an AT and an RT. Each of them have a field named "user_assertion_hash" which matches the cache key. These are indeed 2 separate tokens obtained via OBO. I assume that you've called the front-end app twice.

The 3rd entry is NOT an OBO cache. The cache key is set to ".9188040d-6c67-4c5b-b112-36a304b66dad", which is the ID of the user and the ID of its home tenant (918... is the big MSA tenant). MSAL creates this type of entries if you call AcquireTokenByAuthorizationCode or AcquireTokenSilent. And indeed the tokens inside are not associated with a user_assertion_hash.

Now, 1 hour later, you say "The documents for cache_entry_one, cache_entry_two and cache_entry_three in the cosmos msal container have disappeared. ". MSAL does not set any eviction policies on the L2 cache. What happened to those entries?

In particular I see:

[MsIdWeb] MemoryCache: Read cacheKey yes2cbWRoM3dfQ1XTACHFySTX0t3MPQlnIQNzL8SOEs cache size 0

If this document were present, MSAL would have been able to use it and give your app an access token. This key matches entry number 2.

[SirElTomato]

@bgavrilMS I have not called the front-end app twice, all three entries are from a single authentication.

I have no idea what happens to the three entries, it seems that they just get removed after an hour.

[SirElTomato]

I have just tried using "AddDistributedSqlServerCache" instead of "AddCosmosCache" and it seems to have worked. I will double check this after a longer time period has elapsed tomorrow. Maybe the bug is with the Microsoft.Extensions.Caching.Cosmos package

[bgavrilMS]

@SirElTomato - the first token you cached was for api://<guid>/ScanEmails and then 1 ms later the second one was for "mail.read". The 3rd one (which is not from OBO) came 15ms later also for "Mail.Read".

[SirElTomato]

@bgavrilMS I have just retried using the "AddDistributedSqlServerCache" approach and I am still getting the expiry error. I will send you the logs.