Loading photos for some tenants throws "ProfileStorageUnexpectedException"

[barclayadam]

Describe the bug For a number of tenants we have connected to through an Azure App Registration we are consistently experiencing a "ProfileStorageUnexpectedException" being thrown when loading photos (in a batch).

I cannot find any information about what that actually means.

To Reproduce Create a photo load batch:

using var batchRequestContent = new BatchRequestContent();

foreach (var u in users)
{
    batchRequestContent.AddBatchRequestStep(this._graphServiceClient.Users[u].Photo.Request());
}

Load a response from that batch:

await response.GetResponseByIdAsync<ProfilePhoto>(batchRequestId);

The above line of code throws exception:

Code: ErrorStoreInternalServerError Message: Exception of type 'Microsoft.Fast.Profile.Core.Exception.ProfileStorageUnexpectedException' was thrown. Inner error: AdditionalData: date: 2021-07-22T11:36:17 request-id: a1743be0-6b6f-4bfd-adc7-20ba95dac379 client-request-id: a1743be0-6b6f-4bfd-adc7-20ba95dac379

Expected behavior A photo can be loaded (or a different exception such as NotFound). AB#10354

[andrueastman]

@barclayadam Thanks for raising this issue. Any chance you can provide more information about the issue?

Was the code working before and the issue started recently? Also, just to confirm, is the issue limited to a specific number of tenants while other tenants are okay?

[barclayadam]

Hi @andrueastman

We only have a handful of customers using the app. This is an issue that we have noticed in recent weeks, but have never seen it before when testing our integration.

Hard to say how long the issue has been around for as the tenants we have tested with have never, and still do not, show the issue. Most new tenants also do not show the issue.

We have seen in some instances what appears to be a similar issue, but instead of an exception in the SDK like this, we actually get a .NET YSOD from Graph itself:

For reference, and if it helps to diagnose on the graph service side, that bottom line is: Inner error: AdditionalData: date: 2021-07-23T14:33:53 request-id: 304703a5-5f5e-4b66-8426-85f3205fa2c9 client-request-id: 304703a5-5f5e-4b66-8426-85f3205fa2c9

[andrueastman]

@barclayadam, Thanks for extra information! From the logs, it seems some of the photo requests return as expected with a 200 response or 404 response but one has the unexpected response from the API end.

I have now raised this with the relevant workload team here to investigate and will update the issue here once I get feedback on this.

[barclayadam]

@andrueastman do you have any updates for this issue please?

[andrueastman]

Hey @barclayadam,

Thanks for reaching out again. Just to confirm are you still able to replicate this on your end?

If so, can you share the new request-id for this as the logs do expire so that I can follow up this up for you again.

[barclayadam]

Thank you @andrueastman for quick response.

A few request-id samples from requests that are failing to load user photos are below. It is saying the Code is "UnknownError", so not sure if the same problem as above or not now (we log these differently to stop them crashing the process) but it is happening in the same place. And whether same issue or not, we are seeing a lot of these errors when loading images in batch

Error message example:

Type: Microsoft.Graph.ServiceException
Source: Microsoft.Graph.BatchResponseContent+<GetResponseByIdAsync>d__9`1, Microsoft.Graph.Core, Version=1.21.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
Message: Code: UnknownError
Inner error:
    AdditionalData:
    date: 2021-10-05T06:47:22
    request-id: 5c6e1130-6560-424f-9160-ab1feb4a40db
    client-request-id: 5c6e1130-6560-424f-9160-ab1feb4a40db

Some more request-ids:

`12a925d0-c5fb-48c9-8bd0-46c131690a72`
`6333aa19-5615-41c6-8854-7c1d2c5c2449`
`94777668-8092-4677-981b-b0502d83b60b`

[barclayadam]

In addition to above, we also have many of a different class of error when loading photos:

Type: Microsoft.Graph.ServiceException
Source: Microsoft.Graph.BatchResponseContent+<GetResponseByIdAsync>d__9`1, Microsoft.Graph.Core, Version=1.21.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
Message: Code: UnknownError
Message: {
  "errorCode": "GatewayTimeout",
  "message": "Timed out on external dependency",
  "target": null,
  "details": null,
  "innerError": null,
  "instanceAnnotations": []
}
Inner error:
    AdditionalData:
    date: 2021-10-04T12:37:32
    request-id: b1e952d4-95ed-4b9c-a9dd-c57171dc10dc
    client-request-id: b1e952d4-95ed-4b9c-a9dd-c57171dc10dc

Again, some request-ids:

4f418a54-b554-4d1c-aa81-d805bb60d708 8b9549f8-d0a8-42bd-99e9-9342831023d8

[andrueastman]

Hey @barclayadam,

The feedback given is that the errors experienced are transient and should be resolved now. Are you able to confirm if you are still experiencing this? Is so, is this at high frequency?

[barclayadam]

@andrueastman We are still seeing these issues on a relatively common basis. Some examples:

7160a4e8-a82c-4c6f-a715-7ff1cab81a92 -> "AadGraphCallFailed: Accessing service failed." 4d1a05b3-a041-4823-a6bb-226eef78f23f -> "UnknownError" 3b55a0a1-351a-4135-baf3-7a048e1cfa86 -> "AadGraphCallFailed: Accessing service failed."

Those are 3 from today. We get a relatively large number of these issues every day.

We import often, loading users + groups and user photos. It is only loading user photos that we have any issues with

[petrhollayms]

Thank you for reporting this issue. This appears to be an issue or limitation with the service APIs. Unfortunately, as the Microsoft Graph SDK team, we do not have ownership of the APIs that are causing you issues. We invite you to create a question about the service API to Microsoft Q&A and tagged with one of the [microsoft-graph-*] tags, that way it will get routed to the appropriate team for them to triage:

https://aka.ms/msgraphsupport or directly https://aka.ms/askgraph

For now, we will close the issue on our side but feel free to open it in the relevant repository if you think the issue is specific to SDK. Please let us know if this helps!

Note: We will close this repository on April 19, 2024.