Non-2xx status returned from web service: 400 "TENANT_LICENSE_COUNT_EXCEEDED\"

[likwidtek]

Hey there, I've been working with support but I figured I'd open the issue here as well. I originally was able to use Enable-SpanningUsersFromCSVAdvanced to license about 500 something users but it failed suddenly. I have 219 licenses left to assign, regenerated a new CSV file and ran

PS C:\**********************\4.1.0.0> Import-Module .\SpanningO365
PS C:\**********************\4.1.0.0> Enable-SpanningUsersFromCSVAdvanced -Path "C:\PS C:\**********************\4.1.0.0\import.csv" -UpnColumn 0

And am getting a large error which seems to be indicating I don't have enough licenses. My CSV file contains exactly 219 listings and I have 219 licenses available.

Key parts of the error:

Invoke-WebRequest : {"code":"InternalServerError","message":"Non-2xx **_status returned from web service: 400_**.
HttpResponseError: {\"url\":\"http://bo-a-p-aat.spanning.co/tenant_caches/2398987/users/assign\",\"headers\":{\"cache-c
ontrol\":\"no-store\",\"content-encoding\":\"gzip\",\"content-type\":\"application/json\",\"date\":\"Fri, 30 Jul 2021
20:56:23 GMT\",\"pragma\":\"no-cache\",\"x-spanning-tenant-error\":\"TENANT_LICENSE_COUNT_EXCEEDED\",\"transfer-encodin
g\":\"chunked\",\"connection\":\"Close\"},\"responseBody\":{\"message\":\"Not enough available licenses. No licenses
were assigned.\",\"licensesAvailable\":219,\"users\":

And here:

]},\"r
esponseStatus\":400}"}
At C:\Users\*********************\4.1.0.0\Private\Invoke-SpanningRequest.ps1:191
char:22
+ ...  $results = Invoke-WebRequest -uri $request -Headers $headers -Method ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Screenshot:

[SPMatthewMcD]

Can you send me your file? matt.mcdermott@spanning.com

[SPMatthewMcD]

Or, try trimming 2 off the end and see if we have a zero index error.

[likwidtek]

I'll delete that last 2 records and run again, one moment. I'll also email you the file.

[likwidtek]

Or, try trimming 2 off the end and see if we have a zero index error.

I deleted 2 records and attempted again, same result

[SPMatthewMcD]

After discussing with Engineering this issue is caused by duplicate UPNs. It appears that AAD allows you to recreate UPNs and assigns new MSIDs to them. This is causing a single UPN to return 2 users from the system thus overrunning you licenses when they are this close.

I'll keep this issue open until we fix it on the backend.

[simps100]

Just raised a ticket with support but seems I have the same issue - in the past to licence users i have exported all users to CSV and ran Enable-SpanningUsersFromCSVAdvanced, this would go through skipping those already licenced and applying licence to those who needed it. Seems something has recently changed as this is no longer possible.

Any update on a fix or work around for now other than manually licencing one by one?

Many thanks

[SPMatthewMcD]

What we found is that the when a UPN is recreated in Azure the MSID (GUID) is unique not the UPN. In our system we end up with 2 identical UPNs. This is causing it to appear to be two separate users. We have a fix in QA right now and as long as it passes the tests should release next week.

[SPMatthewMcD]

Also @simps100 we did change the way the module handles bulk assignment, rather than license users one at a time, resulting in 500 requests, we can license up to 500 users in one request. You can still license them one at a time, but the bulk approach is more efficient.

[simps100]

@SPMatthewMcD great thank you good to know - is the process of assigning in bulk the same? Looking to automate this as much as possible :)

[SPMatthewMcD]

The CSV cmdlet just wraps the Enable-SpanningUserList cmdlet for backward compatibility. Enable-SpanningUser uses the Enable-SpanningUserList as well, but only processes one user at a time. So you could use that in a foreach loop to accomplish the task while we get engineering to complete the fix.

[SPMatthewMcD]

Also @simps100 we're planning to release a feature in Q4 that will let you identify an Azure Group for your licensed users. When new users are added they will receive a Spanning License. We'll send a warning email if you run below a threshold that you configure in the product.

[simps100]

@SPMatthewMcD Great thanks will look out for the update hopefully next week - is there an example how to use the Enable-SpanningUser command with a CSV in the meantime?

Good news about using an Azure group to assign licences - thats exactly what we need will be a huge help!

[SPMatthewMcD]

All, during testing we discovered a side effect that may cause the API to return User not found for tenants impacted by this. We're fixing all this in the release.

[SPMatthewMcD]

@simps100 you could try something like (pseudo code let me know if you need more). Assuming you have a "UPN" column in the CSV...

$users = Import-CSV .\users.csv
foreach ($user in $users){
Enable-SpanningUser -UserPrincipalName $user.upn 
}

[simps100]

@SPMatthewMcD Thanks will give that a go - still hoping for a new release this week to address the issues?

[SPMatthewMcD]

We released it yesterday in all regions. Let me know here if it works for you. We'll keep the issue open pending your confirmation.

[simps100]

@SPMatthewMcD thanks do I need to update modules on my local machine or all changes done on your side?

[SPMatthewMcD]

It was done on our side. You might want to try the development branch download to see if the performance is better. I am thinking of making another change to the module since it is consuming so much memory. The JSON conversion is killing the performance. Can you run it with the -verbose flag so you can see the requests in progress?

[simps100]

Hmm seems to work now but i think the behaviour is different to before.

Previously to make sure all users were licenced I'd simply export a list of all our active 365 users - stick them in the spreadsheet and run the command which went through and licenced the ones who needed it ignoring the ones who don't.

If i do that now i still get the licence exceeded error - even though i have over 50 free licences so it seems like its seeing i have 1000+ users in the spreadsheet and not trying to licence any?

Looking forward to this licencing through Azure group will make this 10 times easier for us! :D

[SPMatthewMcD]

True,

It has changed. Previously for a 1000 users we'd make 1000 requests, individually licensing each user. Now we do them in batches of 500 per request.

It would be more efficient to add licenses to unlicensed users, or you can loop through your list and send them one at a time like the module used to do.

Either way, I agree that the Azure Group Based Licensing will help your scenario.

[SPMatthewMcD]

@simps100 This issue should be resolved at this point. We've updated the tenant logic to eliminate the duplication created by recreating a user in AAD.

If this is not working for you please re-open this issue.