`User` creation with `AdditionalData` set fails with `ODataError` "The following extension properties are not available"

Eagle3386 commented 1 month ago

Describe the bug

I'm trying to create users in an Azure B2C tenant via Graph which fails when using the code inside a gRPC web-service, but succeeds in a console POC-style app - using the exact same code snippet?!

Fun fact: setting the 2nd extension property without casting it from uint to int results in the API only complaining about that property, not the other 2.

Expected behavior

Code succeeds, no matter which way its run.

How to reproduce

Code snippet of the actual Graph code:

var user = await graphServiceClient.Users
                                   .PostAsync(new()
                                   {
                                     AccountEnabled = true,
                                     AdditionalData = new Dictionary<string, object>
                                     {
                                       { "extension_{appId}_CustomerIds", "1,2,3" },
                                       { "extension_{appId}_TenantId", (int)1u },
                                       { "extension_{appId}_TenantIds", "4,5,6" }
                                     },
                                     CompanyName = "Test Co.",
                                     DisplayName = "Test Pilot",
                                     GivenName   = "Test",
                                     Identities  =
                                     [
                                       new()
                                       {
                                         Issuer           = "{ourB2Csubdomain}.onmicrosoft.com",
                                         IssuerAssignedId = "quacks@example.net",
                                         SignInType       = "emailAddress",
                                       }
                                     ],
                                     Mail             = "quacks@example.net",
                                     PasswordPolicies = "DisablePasswordExpiration",
                                     PasswordProfile  = new()
                                     {
                                       ForceChangePasswordNextSignIn = false,
                                       Password                      = "Test12345!"
                                     },
                                     PreferredDataLocation = "EUR",
                                     PreferredLanguage     = "de-DE",
                                     Surname               = "Pilot",
                                     UsageLocation         = "DE",
                                   })
                                   .ConfigureAwait(false);

Code snippet for the actual setup in Program.cs of the web-service:

// … code left out for brevity…
var configuration   = builder.Configuration;
var isNonProduction = !builder.Environment.IsProduction();
// … code left out for brevity…
.AddAuthentication()
.AddMicrosoftIdentityWebApi(options =>
  {
    LogCompleteSecurityArtifact = ShowPII = isNonProduction;
    builder.Configuration.Bind(Constants.AzureAdB2C, options);
    options.TokenValidationParameters = new()
    {
      // … code left out for brevity…
    };
  },
  options => builder.Configuration.Bind(Constants.AzureAdB2C, options),
  subscribeToJwtBearerMiddlewareDiagnosticsEvents: isNonProduction)
.EnableTokenAcquisitionToCallDownstreamApi(options =>
{
  // … code left out for brevity…
  options.EnablePiiLogging = isNonProduction;
  options.LogLevel         = isNonProduction ? LogLevel.Always : LogLevel.Info;
})
.AddMicrosoftGraphAppOnly(_ =>
  new(
    new Azure.Identity.ClientSecretCredential(
      configuration[$"{Constants.AzureAdB2C}:{nameof(MicrosoftIdentityOptions.TenantId)}"],
      clientId,
      clientSecret,
      new()
      {
        Diagnostics =
        {
          // … code left out for brevity…
        },
        IsUnsafeSupportLoggingEnabled = isNonProduction
      })))
.AddInMemoryTokenCaches(options =>
  options.AbsoluteExpirationRelativeToNow = TimeSpan.FromDays(isNonProduction ? 14 : 90))
// … code left out for brevity…

SDK Version

5.58

Latest version known to work for scenario above?

AFAICT: none

Known Workarounds

None, because neither using a POC-style console app nor adding those extension properties via PATCH HTTP request is suitable - I expect the Graph API to handle user creation successfully even when extension properties are part of the request's payload.

Debug output

Click to expand log

``` Microsoft.Graph.Models.ODataErrors.ODataError: The following extension properties are not available: extension_{appId}_CustomerIds,extension_{appId}_TenantId,extension_{appId}_TenantIds. at Microsoft.Kiota.Http.HttpClientLibrary.HttpClientRequestAdapter.ThrowIfFailedResponse( HttpResponseMessage response, Dictionary`2 errorMapping, Activity activityForAttributes, CancellationToken cancellationToken) at Microsoft.Kiota.Http.HttpClientLibrary.HttpClientRequestAdapter.SendAsync[ModelType]( RequestInformation requestInfo, ParsableFactory`1 factory, Dictionary`2 errorMapping, CancellationToken cancellationToken) at Microsoft.Kiota.Http.HttpClientLibrary.HttpClientRequestAdapter.SendAsync[ModelType]( RequestInformation requestInfo, ParsableFactory`1 factory, Dictionary`2 errorMapping, CancellationToken cancellationToken) at Microsoft.Graph.Users.UsersRequestBuilder.PostAsync( User body, Action`1 requestConfiguration, CancellationToken cancellationToken) at My.Services.Migration.GraphService.CreateUser(CreateUserRequest request, ServerCallContext context) in D:\Code\Services\My.Services.Migration\GraphService.cs:line 24 ```

Configuration

OS: Windows 11, latest patches & Ubuntu Ubuntu 22.04.4 LTS, latest patches
architecture: x64 (Windows) & x86_64 (Ubuntu)

Other information

I find the Graph SDK docs regarding Azure B2C user properties rather confusing, because the summary of AdditionalData states:

Stores additional data not described in the OpenAPI description found when deserializing. Can be used for serialization as well.

while the one for Extensions states (emphasis by me):

The collection of open extensions defined for the user. Read-only. Supports $expand. Nullable.

Because that suggests 2 things: a) use AdditionalData for reading and writing, but Extensions only for reading - why 2 properties for basically the same amount of information?

andrueastman commented 1 month ago

Thanks for raising this @Eagle3386

From the error log,

The following extension properties are not available:
     extension_{appId}_CustomerIds,extension_{appId}_TenantId,extension_{appId}_TenantIds.

It looks like the extension properties being sent to the API may not be setup. Did you intend to use string interpolation to insert an appId into the property names?

I believe if you use the API documentaion here, you should be able to list the correct names to use for the extension properties and whether they are available in the tenant you are trying to call.

https://learn.microsoft.com/en-us/graph/api/directoryobject-getavailableextensionproperties?view=graph-rest-1.0&tabs=http

Eagle3386 commented 1 month ago

@andrueastman Nope, that's just me, redacting the app ID - in the actual exception, it's extension_abc[…]423_CustomerIds, etc.

Furthermore, rest assured, the extension properties are 100% correct - no doubt about it. That's even confirmed by what I wrote initially:

(…) but succeeds in a console POC-style app - using the exact same code snippet (…)

Here's even a screenshot from our Azure B2C tenant's User attributes blade to confirm proper naming: {073DD721-399A-40A0-ADB5-6236F4E4617D}

And using the Graph API you've linked to, I get this:

For convenience, the relevant columns as text: DataType	IsMultiValued	Name	TargetObjects	DeletedDateTime	AdditionalData	BackingStore
`String`	`False`	`extension_abc[…]423_TenantIds`	`[User]`	`""`	`[]`	`InMemoryBackingStore`
`Integer`	`False`	`extension_abc[…]423_TenantId`	`[User]`	`""`	`[]`	`InMemoryBackingStore`
`String`	`False`	`extension_abc[…]423_CustomerIds`	`[User]`	`""`	`[]`	`InMemoryBackingStore`

So, I already confirmed that everything is set up as required, yet the API complains where it shouldn't even try to start & simply do what it's ordered to: create a user with additional data / extension properties.

andrueastman commented 1 month ago

Thanks for coming back to this @Eagle3386

When performing the request using the console app vs the service, do they use the same appId and permissions? Are you also by any chance able to make the request successfully on Graph Explorer?

Eagle3386 commented 1 month ago

@andrueastman

Yes, of course they do!

Besides, how could I possibly get The following extension properties are not available upon user creation, yet querying for all extension properties does return a non-empty collection, if the permissions aren't sufficient?

Here's the POC app's relevant code snippet as kinda proof:

  private static GraphServiceClient AppClient { get; set; }

  private static ClientSecretCredential Credential { get; set; }

  private static Settings Settings { get; set; }

  public static async Task<string> CreateUserAsync()
  {
    var user = await AppClient.Users
                              .PostAsync(new()
                              {
                                AccountEnabled = true,
                                AdditionalData = new Dictionary<string, object>
                                {
                                  { "extension_abc[…]423_CustomerIds", "1,2,3" },
                                  { "extension_abc[…]423_TenantId", (int)1u },
                                  { "extension_abc[…]423_TenantIds", "4,5,6" }
                                },
                                CompanyName = "Test Co.",
                                DisplayName = "Test Pilot",
                                GivenName   = "Test",
                                Identities  =
                                [
                                  new()
                                  {
                                    Issuer           = "{ourB2Csubdomain}.onmicrosoft.com",
                                    IssuerAssignedId = "quacks@example.net",
                                    SignInType       = "emailAddress",
                                  }
                                ],
                                Mail             = "quacks@example.net",
                                PasswordPolicies = "DisablePasswordExpiration",
                                PasswordProfile  = new()
                                {
                                  ForceChangePasswordNextSignIn = false,
                                  Password                      = "Test12345!"
                                },
                                PreferredDataLocation = "EUR",
                                PreferredLanguage     = "de-DE",
                                Surname               = "Pilot",
                                UsageLocation         = "DE",
                              })
                              .ConfigureAwait(false);
    user = await AppClient.Users[user.Id]
                          .GetAsync(configuration => configuration.QueryParameters.Select =
                          [
                              "extension_abc[…]423_CustomerIds",
                              "extension_abc[…]423_TenantId",
                              "extension_abc[…]423_TenantIds",
                              "GivenName",
                              "Id",
                              "Identities",
                              "Surname"
                          ])
                          .ConfigureAwait(false);
    return $"Name:         {user.GivenName} {user.Surname
      }\nMail:         {user.Identities![0].IssuerAssignedId
      }\nID:           {user.Id
      }\nCustomer IDs: {string.Join(", ", user.AdditionalData.First(data => data.Key.EndsWith("CustomerIds")).Value)
      }\nTenant ID:    {user.AdditionalData.First(data => data.Key.EndsWith("TenantId")).Value
      }\nTenant IDs:   {string.Join(", ", user.AdditionalData.First(data => data.Key.EndsWith("TenantIds")).Value)}";
  }

  public static async Task<string> GetAppOnlyAccessTokenAsync() =>
    (await (Credential ?? throw new NullReferenceException("Graph uninitialized for app-only auth."))
        .GetTokenAsync(new TokenRequestContext([ "https://graph.microsoft.com/.default" ]))).Token;

  public static void InitializeGraphForAppOnlyAuth(Settings settings)
  {
    Settings   =   settings ?? throw new NullReferenceException($"{nameof(Settings)} cannot be null.");
    Credential ??= new(Settings.TenantId, Settings.ClientId, Settings.ClientSecret);
    AppClient  ??= new(Credential, [ "https://graph.microsoft.com/.default" ]);
  }

… which returns:

User created:
Name:         Test Pilot
Mail:         quacks@example.net
ID:           d0a9aeb0-2a4a-47eb-83fa-c79d89fae676
Customer IDs: 1,2,3
Tenant ID:    1
Tenant IDs:   4,5,6

Using that snippet's code for retrieval inside the actual web-service, the user's AdditionalData property is populated with exactly one entry:

Key: @odata.context
Value: https://graph.microsoft.com/v1.0/$metadata#users(extension_abc[…]423_CustomerIds,extension_abc[…]423_TenantId,extension_abc[…]423_TenantIds,givenName,id,identities,surname)/$entity

… without any exception, so querying works, just as the extension properties are there.

So, can we finally switch from "Where's the error in my code?" to "Where's the bug in Graph?" Because up to now, it seems we're trying pretty much everything to enforce the impression of the former instead of locating the latter.

For example, you could tell me how I can enable Graph Explorer to connect to our B2C tenant, because it pretty much only connects to our workforce one.

Lastly, rest assured that I already tried the user creation via Graph Explorer & it perfectly created the user, including the extension properties - though, as already stated, it creates them in the workforce tenant which is obviously not what's desired.

andrueastman commented 1 month ago

Are you able to capture the request-id and client request id from the error when it fails? Using this, we can file an issue with the API team to understand what could be wrong in this scenario.

You should be able to use the guidance here and retrieve the properties from innerError

https://github.com/microsoftgraph/msgraph-sdk-dotnet/blob/main/docs/errors.md#handling-errors-in-the-microsoft-graph-net-client-library

Eagle3386 commented 1 month ago

Yes, of course! Thanks for asking, @andrueastman! 👍🏻

There you go:

Error.InnerError.RequestId: fae3c201-8215-45b3-89aa-f81b5139ea5a
Error.InnerError.ClientRequestId: 16077f71-69b0-42ed-984f-56054a577dc2

andrueastman commented 1 month ago

Thanks for this. I've raised an issue with the workload and will give feedback based on what they find.

https://portal.microsofticm.com/imp/v5/incidents/details/549539260/summary

Eagle3386 commented 1 month ago

Thanks so much, @andrueastman! 👏🏻 Awaiting their/your feedback now.

andrueastman commented 1 month ago

@Eagle3386 Feedback from the API team is that the string interpolated here is incorrect. extension_{appId}_CustomerIds.

It looks like you may be interpolating the tenant_id instead instead of the app_id in the failing scenario. From their perspective, the extension attributes exist but the request has the incorrect id that look like the tenant id.

Any chance you can double check to confirm?

Eagle3386 commented 1 month ago

@andrueastman please report precisely back to them:

You're wrong, in multiple ways & right from the start:

There's no string interpolation happening - at all! This is even backed as my latest log - which should've been provided to you instead of the older one to begin with - clearly mentions the shortened extension_abc[…]423_ instead of any _{appId}_.
When ignoring this fact, there's no interpolation of our Azure tenant's ID whatsoever, but instead simply one (and only one!) extension property whose name ends with tenantId.
Even ignoring this fact, too, there's still one ultimate fact: not only is the actual GraphServiceClient's code for creating the user identical in both executables, but is any used ID (Azure app/client ID, Azure tenant ID, etc.), too!

So, once again: I did double checke - Again! - yet, the error remains the same - just like the code remains to comply with both, the API & its docs, too.

Now please, with all due respect as I don't feel you're taking this seriously: rather do debug the given ClientRequestId / RequestId in your backend at least once than trying to blame my code.

andrueastman commented 1 month ago

@Eagle3386

Apologies for any frustrations here. I may have not shared enough details before. The team has indeed debugged the request id.

The request id shared above here has the following details from the back-end logs. The API team has also confirmed the extension properties do not exist in the given tenant based of the information coming in from the request.

App ID : 682xxxxxxxxxxxx032 Tenant ID : f677xxxxxxxxxxxx0668f

The received request on the backend logs shows the error as where the id matches the tenant id and not the app id The following extension properties are not available: extension_f677xxxxxxxxx668f_CustomerIds.

Any chance you can confirm if this info is incorrect from your end? If not I can feedback any inconsistencies back to them...

Eagle3386 commented 1 month ago

@andrueastman

First of all, regarding this:

Apologies for any frustrations here. I may have not shared enough details before. The team has indeed debugged the request id.

Apologies accepted, no hard feelings & thanks for confirming actual debugging!

Now, let's get back to work:

The request id shared above here has the following details from the back-end logs. The API team has also confirmed the extension properties do not exist in the given tenant based of the information coming in from the request.

App ID : 682xxxxxxxxxxxx032 Tenant ID : f677xxxxxxxxxxxx0668f

I can confirm, both are correct - see the app registration's overview as proof:

The received request on the backend logs shows the error as where the id matches the tenant id and not the app id The following extension properties are not available: extension_f677xxxxxxxxx668f_CustomerIds.

Any chance you can confirm if this info is incorrect from your end? If not I can feedback any inconsistencies back to them...

I can partly confirm this, but by all means, I truly apologize for messing both ID values up first! Though, after fixing them, i.e., using app ID instead of tenant ID (which I seem to have messed up due to some docs asking to use it as AzureAdB2C:Domain's value, too), I'm still receiving the very same error:

The following extension properties are not available:
extension_682[…]032_CustomerIds,extension_682[…]032_TenantId,extension_682[…]032_TenantIds.

Error.InnerError.RequestId: 2e785e06-f1db-43d9-ad24-095a9077c1f6 Error.InnerError.ClientRequestId: cd4e34ba-6f89-4b01-9818-e022eca11184

Please, can you check what's still causing the error?

Additionally, I really want to know how this can actually work in the POC program, but only the actual service complains about the wrong app ID?! 😳😅

andrueastman commented 1 month ago

Thanks for the extra info here. I've fed this new request id back to the API team issue request and will give an update on any new findings from their point of view.

Eagle3386 commented 1 month ago

Again, thanks so much, @andrueastman! 👏🏻 Eagerly awaiting their/your feedback.. 😉

andrueastman commented 1 month ago

@Eagle3386 We've received feedback from the API teams.

According to the request, the App ID for the request is 682xxxxxxxxxxxx032 and the Tenant ID is f677xxxxxxxxxxxx0668f and the request makes a request to an extension property with the format below extension_f677xxxxxxxxxxxxxxx0668f_CustomerIds.

The API team gives the feedback that the extension properties in this tenant are setup with another AppId that has the format f67730xxxxxxxxxxxxxxxxxa562a.(It suspiciously looks like the other tenant Id at the first few characters.). So the valid extension properties should look like this for that tenant extension_f67730xxxxxxxxxxxxxxxxa562a_CustomerIds.

According to the team, the AppId that should be used is the AppId that is used to configure the extension properties in the tenant and not the AppId that makes the request. https://learn.microsoft.com/en-us/graph/extensibility-overview?tabs=http#directory-microsoft-entra-id-extensions

Would it be possible to confirm if this is the information is consistent from your end?

Eagle3386 commented 1 month ago

@andrueastman You're the man - and the API team as well, thanks so much! ❤️

It's the b2c-extensions-app. Do not modify. Used by AADB2C for storing user data.'s app ID that needs to be used - which its name already suggests. Well, I've got to admit: this leaves me pretty embarrassed.. 🙈😅

Some more background/context: I always compared only the begin & end of the ID & since they start equally & end similarly, I've obviously missed the slight difference - in my POC console app, the extension property names are hard-coded while the service code reads it from the AppSettings.json file.

Lastly, since I fail to find the proper paragraph within the docs: what to change inside my custom policy so that it includes the extension properties as claims within the AccessToken?

andrueastman commented 1 month ago

@Eagle3386

Lastly, since I fail to find the proper paragraph within the docs: what to change inside my custom policy so that it includes the extension properties as claims within the AccessToken?

I believe the correct links for the documentation of that scenario would be the following depending on what you're trying to achieve.

Eagle3386 commented 1 month ago

@andrueastman Thanks for both links, but I'm still struggling with your help:

First of all: since Azure B2C isn't converted into the Entra "naming scheme", yet, are the docs you've linked to still correct/relevant? Because I followed this one: https://learn.microsoft.com/en-us/azure/active-directory-b2c/user-flow-custom-attributes?pivots=b2c-custom-policy - yet, no claim containing any of the 3 extension attributes is included in the access token issued to the user!? 😞

microsoftgraph / msgraph-sdk-dotnet