Azure Stack: Sign-in broken for guest accounts, and discovery process "intractable"

When adding a custom environment (Azure Stack) the storage explorer application assumes the login account used should be used with the home directory. This is not correct, and will result in the following error:

{
  "message": "\"Unexpected token format: urn:ietf:wg:oauth:2.0:oob?error=access_denied&error_description=AADSTS65005%3a+Invalid+resource.+The+client+has+requested+access+to+a+resource+which+is+not+listed+in+the+requested+permissions+in+the+client%27s+application+registration.+Client+app+ID%3a+04b07795-8ddb-461a-bbee-02f9e1bf7b46.+Resource+value+from+request%3a+https%3a%2f%2fadminmanagement.msazurestack.onmicrosoft.com%2f9f4c9261-e5d2-4e83-bc6c-763f54de5046.+Resource+app+ID%3a+42547758-86d7-45f3-81f8-183181e67712.+List+of+valid+resources+from+app+registration%3a+.%0d%0aTrace+ID%3a+7428aebd-5a29-451c-aae4-611389842b00%0d%0aCorrelation+ID%3a+7ecb73fb-6508-4f0c-abad-d4987af5e641%0d%0aTimestamp%3a+2018-12-11+23%3a36%3a42Z\""
}

Above happens when you use an account which is only a guest in the "home" directory of an Azure Stack. The storage explorer code assumes they can use the common login endpoint and acquire a token for the target ARM application in that directory. This is a permission that has to be manually granted when a directory is onboarded to Azure Stack (we create permission grants for Azure PowerShell and Azure CLI applications to acquire a token against the specific ARM instance, and it looks like Storage Explorer is using the app ID for Azure CLI).

Additionally, the discovery process used is extremely inefficient and resulted in over 40K+ requests in the background, consuming 100% CPU and only terminating when the access tokens expired, which caused it to restart the process over again. When a custom environment is added, there should not be any enumeration of SPs in the authenticated directory, and instead only the ARM resource as advertised at the metadata endpoint should be used to acquire a token, and then subsequently call the /subscriptions API at that endpoint.

Below image shows how tokens are repeatedly acquired for resources in many different directories and tenant details are queried. This process essentially continues forever consuming maximum resources in the background. While this is occurring, the accounts tab has a spinning icon indicating to the user it is trying to load their subscriptions. To seems to have been developed with accounts that do not belong to many directories, and to directories without many Azure Stack instances registered to them. storageexplorercalls

I'll address each thing 1 by 1:

Guest accounts for Azure Stack Can you expand on this? Or I can try seeing if I understand. We enumerate SPs (applications) looking for Azure Stacks. When we find one, we try to get its endpoints. From there, we get a login endpoint, which we eventually use as the resource for getting tokens for doing things like listing resources. You're saying that if an account is a guest, then it may not be able to use that endpoint, right?
Enumerating SPs for custom environments You're probably right that we could assume that we don't need to enumerate SPs for accounts added via a custom environment. This problem should be fairly easy to solve, though we might need to think about it a bit more/figure out if we need to at least expose the ability.
This feature being developed with accounts that don't have a lot of Azure Stack instances Yes that is true! 😄 That being said, we knew there might be people with lots of Azure Stack instances on their account, but we wanted to wait for feedback that such people really existed before trying to solve the problem. I believe you're the first person to let us know that due to such a situation, loading your subscriptions is taking a reallllllly long time. We'll start thinking about how best to solve the problem. Do you have any initial thoughts on what you might like to see from the user experience standpoint?

Regarding guest accounts; the flow looks like this:

Open latest version of storage explorer in clean environment.
Select "Add an account".
Choose "custom environment" and provide the ARM endpoint of your Azure Stack.
Choose "sign in" and use an account which is a guest user in the "home" directory of Azure Stack.
Upon sign-in, Storage Explorer will render an error message (described in my previous post).

The problem above comes from the fact that Storage Explorer is using the "common" login endpoint in Graph. This is a common problem in first-party services built by Microsoft, as they "assume" they can authenticate to any directory. This not true for "third-party" applications, which need to be installed and have permissions granted in each directory (or when Microsoft first-party applications are attempting to interact with third-party applications, like in the case of Azure Stack).

The primary issue is that when you use the /common login endpoint, AAD will "pick" the user's home tenant as the directory tenant in which to authenticate. But the user's home directory is not the one that is setup to work with Azure Stack (in this example).

Every tool that wants to sign people into AAD needs to allow a way to "advertise" the target directory into which they would like to authenticate. Even Azure Portal has such a feature: when you navigate to Portal.Azure.com, it will use the "common" login endpoint which will sign the user into their home directory be default. This isn't a problem for Azure, as both the client (Portal) and the resource (ARM) are first-party applications that can automatically install themselves into the user's home directory, and have their permissions granted. They can then subsequently call special first-party APIs that tell them all the directories into which the user could successfully authenticate, and render a "directory tenant picker" in the upper-right for the user to change which directory they are working in.

The above flow obviously won't work for third-party applications, and even though it isn't strictly necessary for the Azure scenario, the portal allows the user to append the target directory name or tenant ID to the first segment of the portal URL to advertise the target directory. So if you navigate to Portal.Azure.com/ageofempiresonline.com this will actually sign you into the Microsoft directory (if you have a Microsoft account) since you can use either the tenant ID or ANY verified domain name associated to the target directory. This mechanism is critical for Azure Stack to allow users to change which directory they authenticate with.

We need a similar mechanism in Storage Explorer to allow the tool to be used with Azure Stack. I have to run to a meeting now but I will reply with more details about "target user experience" as there is significant overlap here with the "discovery process" issues mentioned in my original post.

Just to make sure that you have targeted Storage Explorer for Azure Stack by using the Edit->Target Azure Stack APIs menu, correct?

      Just to make sure that you have targeted Storage Explorer for Azure Stack by using the Edit->Target Azure Stack APIs menu, correct?

Yes I have switched to target. I'm preparing more details about the discovery process, but I'm having trouble aborting it as it seems to be stuck in an infinite loop making the same Graph calls, and when I select "remove account" it looks like it is waiting for the discovery process to complete before I can remove it :(

Regarding enumerating SPs for custom environments:

It's a little bizarre why this is here at all, when the custom environment is provided explicitly. There shouldn't need to be anything to "discover" here (though you could build a separate Azure Stack discovery process where user doesn't have to enter anything; we can discuss that later).

Open latest version of storage explorer in clean environment.
Select "Add an account".
Choose "custom environment" and provide the ARM endpoint of your Azure Stack.
Choose "sign in" and use an account which is NOT a guest user in the "home" directory of Azure Stack.
I didn't see this call in fiddler, maybe it was cached somewhere from a previous run, but something has enumerated all the SPs in the directory to "detect" "all" the "Azure Stack Resource Manager" SPs.
I see a call to the /tenants API on the Azure Stack ARM. This API just returns the tenant which the call user is authenticated to (we cannot call the private first-party API to list all the tenants to which the authenticated user belongs).
Storage Explorer begins acquiring tokens for thousands of SPs in this directory. Eventually this finishes using an account which only "belongs" to a single directory, and I see that it found 3 subscriptions in the custom environment I added. I don't see the call to /subscriptions in fiddler, though, nor do I see any calls made with any of the thousands of tokens retrieved for all these various SPs.

The above flow is unusual because the user has already given you their Azure Stack API endpoint. You can query the metadata endpoint and get the audience field which it looks like you do, allowing you to immediately get a token and call /subscriptions API at that endpoint. You never need to look-at or considering anything to do with any SPs in any directories.

Now, as mentioned before, if you want to have a discovery process for multiple Azure Stack environments, or multiple directories for a single added Azure Stack, this would require some changes. One possibility, would be to call the /tenants API on the Azure ARM of the corresponding region to "discover" all the possible tenants into which the designated user can authenticate. This gets you the list of tenants that you "need" to check. You can then decide if you just want to check the target custom environment (the single ARM audience from the single endpoint the user provided) in each of these directories, or if you want to look for other Azure Stack SPs besides the current one indicated when adding the custom environment. I would advise that you stick to the specific environment, or you'll have a discovery process that won't work for many of our test directories which are filled with "dead" SPs.

Once you have the list of tenants, and let's say you are sticking to the single ARM audience field you've discovered when the custom environment was added, you now just need to attempt to acquire a token for that audience in each of the discovered tenants, and call /subscriptions against the ARM endpoint in each of these tenants. Now, some of these tenants, you will fail with a 403 or 400 response from Graph, as the applications and permissions will not be enabled in the user's directory. In some cases, if the user is an admin, there might be some consent objects created that will allow the token to be acquired, but the subsequent call to ARM would be a 403 as ARM only authorizes calls to known directories which have been onboarded.

Hey @keystroke , thank you for the thorough replies! You answer about the common endpoint is something that I've been trying to get information on for a long time. I think I agree with most everything you're suggesting/saying, and I definitely want to get the guest tenant situation figured out. It is a scenario the Azure Stack team has wanted us to support, but they never quite explained what we need to do to support it (though you've explained it quite nicely 😄 ). We'll try to get this done as soon as possible, but thank you in advance for your patience as I'm not sure how soon that will be. I may post on this issue in the future with questions, and if you'd like, I can also reach back out once we have something for you to try out.

PS: sorry for not replying to you sooner, I was on vacation. ✈️

@MRayermannMSFT I was also on vacation so no worries! Of course, please feel free to reply with further questions / comments or to reach-out internally!

I started typing this out but lost all my progress due to machine reboot, so apologies if this is more terse the second time around :)

As an example of a full discovery process, I've shared below which shows a PowerShell implementation. Before jumping-in, I want to call-out that Azure Stack also supports ADFS as an identity system, instead of AAD, and there are some differences associated to that. But more on that later.

We begin by authenticating the user using the common login endpoint to their home directory. This is simulating using my PS client and the known home directory of the test user in this example:

# Authenticate to the "home" directory of the user
Import-Module D:\One\AzureStack\Solution\Deploy\src\CloudDeployment\Roles\IdentityProvider\GraphAPI.psm1
Initialize-GraphEnvironment -DirectoryTenantId msazurestack.onmicrosoft.com -PromptForUserCredential

Next, we authenticate with Azure Resource Manager (in the same region of our chosen login endpoint) to discover all the directory tenants to which the user can authenticate:

# Call the Azure Resource Manager (in the same region as the AAD endpoint used) to discover the tenants to which the user belongs
$azureResourceManagerAudience = (Invoke-RestMethod https://management.azure.com/metadata/endpoints?api-version=2015-01-01).authentication.audiences[0]
$azureResourceManagerToken    = (Get-GraphToken -Resource $azureResourceManagerAudience -UseEnvironmentData).access_token
$directoryTenantIds           = @((Invoke-RestMethod https://management.azure.com/tenants?api-version=2015-01-01 -Headers @{Authorization="Bearer $azureResourceManagerToken"}).value.tenantId)

As you can see, we've found a lot of directories for this user:

C:\> $directoryTenantIds | measure

Count    : 19
Average  : 
Sum      : 
Maximum  : 
Minimum  : 
Property :

The next step is to find all the "candidate" Azure Stack API SPs; we'll focus on the first directory for now, but remember that we'll need to repeat the following steps for each directory (and the same Azure Stack environment could appear in multiple directories due to multitenancy features of Azure Stack).

# Focus on the first directory for now
$directoryTenantId = $directoryTenantIds[0]

# Find service principals in the directory which are advertised as "Azure Stack Resource Manager applications"
$startTime           = Get-Date
$servicePrincipals   = (Find-GraphApplicationDataByServicePrincipalTag -StartsWith "MicrosoftAzureStack" -SkipApplicationLookup -Verbose).ServicePrincipal
$possibleAzureStacks = $servicePrincipals | Group HomePage | Sort Count
$endTime             = Get-Date

Taking a look at the returned data: this took 23 seconds for the first directory (using a slow language, admittedly) and we discovered 262 unique possible Azure Stack endpoints, however some of those endpoints have hundreds of SPs associated to them. This is another caveat to mention - Azure Stack environments are often setup in isolated networks and can use the same DNS names in these isolated networks. If the user changes which network they are on, they might be communicating with an entirely different Azure Stack! So we need to make sure that endpoint / DNS name is NOT the identifier for an Azure Stack, but rather the SP oid, appId, or audience is what unique identifies an Azure Stack environment.

C:\> [math]::Round(($endTime - $startTime).TotalSeconds)
23

C:\> $possibleAzureStacks | Select Count,Name -Last 5

Count Name                                                
----- ----                                                
  585 https://adminmanagement.east.azurestack.contoso.com/
  587 https://management.east.azurestack.contoso.com/     
  860 https://adminmanagement.local.azurestack.external/  
  886 https://api.azurestack.local/                       
  982 https://management.local.azurestack.external/       

C:\> $possibleAzureStacks | measure

Count    : 262
Average  : 
Sum      : 
Maximum  : 
Minimum  : 
Property :

So now that we have all the possible Azure Stacks, we need to see which of those endpoints are actually reachable, and of the reachable endpoints, to which SP the responding service is associated, and thus, which Azure Stack we are talking to.

# Filter currently-unreachable Azure Stack endpoints
$startTime   = Get-Date
$azureStacks = $possibleAzureStacks | Where { ($d = try { [System.Net.Dns]::GetHostAddressesAsync(([Uri]$_.Name).Host).GetAwaiter().GetResult() } catch {}) -ne $null }
$endTime     = Get-Date

Checking the resolved data, it took 70 seconds to test all the endpoints serially, and we filtered down from 262 endpoints to 17 which are actually reachable on my machine (many of the other endpoints are just "dud" SPs for environments which no-longer exist, but the SPs were not cleaned-up).

C:\> [math]::Round(($endTime - $startTime).TotalSeconds)
70

C:\> $azureStacks | measure

Count    : 17
Average  : 
Sum      : 
Maximum  : 
Minimum  : 
Property :

Alright, now we know what is next, we need to check each of these 17 endpoints to see if we can authenticate, and if we have any subscriptions in any of them. Since there are only 17, I just did this serially for this example, shown below:

$data = @()
$startTime = Get-Date
foreach ($azureStack in $azureStacks)
{
    $ErrorActionPreference = 'Stop'
    try
    {
        $azureStackResourceManagerEndpoint = $azureStack.Name.TrimEnd('/')
        $azureStackResourceManagerAudience = (Invoke-RestMethod "$azureStackResourceManagerEndpoint/metadata/endpoints?api-version=2015-01-01" -TimeoutSec 15).authentication.audiences[0]
        $azureStackResourceManagerToken    = (Get-GraphToken -Resource $azureStackResourceManagerAudience -UseEnvironmentData -Verbose:$false).access_token
        $azureStacksubscriptions           = Invoke-RestMethod "$azureStackResourceManagerEndpoint/subscriptions?api-version=2015-01-01" -Headers @{Authorization="Bearer $azureStackResourceManagerToken"} -TimeoutSec 15

        $data += [pscustomobject]@{
            TenantId      = $directoryTenantId
            Endpoint      = $azureStackResourceManagerEndpoint
            Audience      = $azureStackResourceManagerAudience
            Subscriptions = $azureStacksubscriptions.value | Select subscriptionId, tenantId, displayName
        }
    }
    catch
    {
        Write-Warning "Failed to resolve subscriptions from endpoint '$azureStackResourceManagerEndpoint': $_"
    }
}
$endTime = Get-Date

And when we check the results; it took just under 2 minutes to grab all the subscriptions we found in these environments. I'm only showing the subscription display names, bit hopefully this illustrates the process you'd use to discover all the Azure Stack environments and subscriptions of a user in a specific directory.

C:\> [math]::Round(($endTime - $startTime).TotalSeconds)
110

C:\> $data | measure

Count    : 6
Average  : 
Sum      : 
Maximum  : 
Minimum  : 
Property : 

C:\> $data | Sort Endpoint | Select Endpoint,@{N='Subscriptions';e={$_.Subscriptions.displayName}} | ft -AutoSize -Wrap

Endpoint                                                                  Subscriptions                                                                                                        
--------                                                                  -------------                                                                                                        
https://adminmanagement.northwest1.azurestack.selfhost.corp.microsoft.com {Default Provider Subscription, Metering Subscription, Consumption Subscription}                                     
https://adminmanagement.northwest3.azurestack.selfhost.corp.microsoft.com {Default Provider Subscription, Metering Subscription, Consumption Subscription}                                     
https://adminmanagement.nw4.azurestack.selfhost.corp.microsoft.com        {Default Provider Subscription, Metering Subscription, Consumption Subscription}                                     
https://management.northwest1.azurestack.selfhost.corp.microsoft.com      {SC-3dbe0cb4-14fd-4b4a-9b6a-b5cda8b00f6c, SC-e94b9ae0-d9a8-4056-89f9-2cb4b27e189d,                                   
                                                                          SC-e2f22355-74a7-40aa-905b-cf8abd0cbe92, SC-79451949-e786-4846-a124-b5afca3c1076}                                    
https://management.northwest3.azurestack.selfhost.corp.microsoft.com      RCPortal Public Offer                                                                                                
https://management.nw4.azurestack.selfhost.corp.microsoft.com

One final thing to note, is that you'll want to repeat the above flow for each of the directory tenants we discovered back at the beginning, and when you do, you may find that some of the SPs appear in more than one of the directories (I briefly mentioned this before). This is because additional directories can be onboarded to a single Azure Stack environment, and a user could be added as a guest to more than one directory, and we have some customer who use this kind of setup which results in a single user having subscriptions in multiple directories. In the discovery process we've "implemented" above, we would want to associate the user subscriptions in different directory tenants, but for the same Azure Stack environment.

And a last note, to show the kind of failures we got when enumerating through these 17 environments:

WARNING: Failed to resolve subscriptions from endpoint 'https://management.angel01.azurestack.direct': An error occurred while trying to make a graph API call: {"error":"invalid_grant","error_description":"AADS
TS65001: The user or administrator has not consented to use the application with ID '1950a258-227b-4e31-a9cf-717495945fc2' named 'Microsoft Azure PowerShell'. Send an interactive authorization request for this 
user and resource.\r\nTrace ID: 5992e660-4613-4082-994a-b8a489430f00\r\nCorrelation ID: 06b86f09-b2f1-4d62-9a25-3a09233f5168\r\nTimestamp: 2019-01-09 04:04:01Z","error_codes":[65001],"timestamp":"2019-01-09 04:
04:01Z","trace_id":"5992e660-4613-4082-994a-b8a489430f00","correlation_id":"06b86f09-b2f1-4d62-9a25-3a09233f5168","suberror":"consent_required"}

WARNING: Failed to resolve subscriptions from endpoint 'https://api.d2-h01c2.masd.stbtest.microsoft.com': The operation has timed out.

WARNING: Failed to resolve subscriptions from endpoint '': Invalid URI: The hostname could not be parsed.

Hopefully that was helpful / informative and showcases a bit of the approach for a discovery process. BUT, you may decide this kind of discovery process should not be used, and instead have the user give their "API endpoint" and you just call /metadata/endpoints API to get the audience, and then /subscriptions (after acquiring a token) and leave it at that; or PERHAPS, use part of the discovery process to call Azure API endpoint to get the list of tenants for that user, and then try to authenticate to this single ARM instance using each of those tenants (many of which will likely fail with the warnings shown above) which will allow you to discover all the subscriptions associated to that Azure Stack environment, and you can still use the /common login endpoint for this initial flow, because your first call will be to Azure API endpoint which can successfully authenticate in the user's home directory.

Please let me know if you have further questions or want to chat!

Any update on this? Storage explorer is completely unusable on my machine even for connecting to Azure, as it blocks at the splash screen and consumes 100% CPU making endless graph queries in the background.

microsoft / AzureStorageExplorer

Azure Stack: Sign-in broken for guest accounts, and discovery process "intractable" #995