[FEATURE REQ] Support TokenCredential in Azure Files SDK (ShareClient, etc.)

[jstutson-rel]

Library name

Azure.Storage.Files.Share

Please describe the feature.

Resubmitting closed feature request https://github.com/Azure/azure-sdk-for-net/issues/17000.

The ShareClient of Azure.Storage.Files.Share should support the DefaultAzureCredential. Or another means of retrieving current size and quota for a file share should be provided that supports TokenCredential.

var client = new ShareClient( new Uri("https://myaccountname.file.core.windows.net/mysharename"), new DefaultAzureCredential());

Look at the equivalent flow in powershell. No shared access key is needed.

#Connect to Azure and Import Az Module
Connect-AzAccount -Identity
Import-Module -Name Az.Accounts
Import-Module -Name Az.Storage
Set-AzContext -SubscriptionId $subscription_id

# Get file share
$StorageContext = New-AzStorageContext -StorageAccountName $storage_account_name -Anonymous
$PFS = Get-AzRmStorageShare -ResourceGroupName $resource_group -StorageAccountName $storage_account_name -Name $file_share_name -GetShareUsage

# Get provisioned capacity and used capacity
$ProvisionedCapacity = $PFS.QuotaGiB
$UsedCapacity = $PFS.ShareUsageBytes

The object returned by Get-AzRmStorageShare may be quite different from a ShareClient, but it does provide the size and quota information commonly needed for quota monitoring. This is an extremely common need for Azure Files and it's already perplexing enough that this kind of auto-expansion logic isn't already built-in to the platform on top of needing to make two API calls to get the size and quota (GetStatisticsAsync and GetPropertiesAsync) vs the single line in Powershell.

The reason for not supporting TokenCredentials with ShareClient in the prior feature request was that SMB protocol doesn't support it. Does getting size and quota actually require SMB protocol? If not, it would be good to get a separate client that can be used to retrieve this information using the DefaultAzureCredential().
If it does need SMB, then how is powershell pulling this off? It would seem it's doing some work for us under the covers.

[github-actions[bot]]

Thank you for your feedback. This has been routed to the support team for assistance.

[PramodValavala-MSFT]

@jstutson-rel Thanks for the feedback! We have routed this to the owning team to review.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[jstutson-rel]

Hello, any word back on this? It's actually looking potentially worse than I thought. The share(s) Azure File Shares I am accessing all require virtual network peering to access. This is really only meant to apply to actual file access, and again, I only need to get the current usage and size quota properties and update the quota if necessary. The network peering really should not apply, and I don't think it does when accessing via AZ CLI.

If I have to peer my function app with these shares just to get size and quota, I'm going to have to upgrade from the consumption plan to the premium plan. This is not good.

[amnguye]

Hi,

Is the issue that this line of code isn't working for you? Because ShareClient and ShareFileClient do support TokenCredential currently (e.g. 12.14.0-beta.1 and above).

var client = new ShareClient( new Uri("https://myaccountname.file.core.windows.net/mysharename"), new DefaultAzureCredential());

I didn't see an attached line of code that says what API call you're making with the client that's returned. Could you return the stack trace that is occuring from the API, along with which API you're calling?

[jstutson-rel]

I am using Azure.Storage.Files.Shares v12.14.0

                Response<ShareStatistics> stats = await shareClient.GetStatisticsAsync().ConfigureAwait(false);
                shareUsageBytes = stats.Value.ShareUsageInBytes;
                int shareUsageGiB = (int)Math.Ceiling((double)shareUsageBytes / 1024 / 1024 / 1024);
                Response<ShareProperties> shareProperties = await shareClient.GetPropertiesAsync().ConfigureAwait(false);
                shareQuotaGiB = shareProperties.Value.QuotaInGB ?? 0;

and if those calls succeeded, we'd set the quota later: await shareClient.SetQuotaAsync(safeTargetQuotaGiB).ConfigureAwait(false); As mentioned, I am retrieving share usage and share quota to ensure adequate headroom. Are there other APIs that can retrieve this information and use token creds? We're using user managed identity for this function, and again, this integrated security approach works perfectly fine to retrieve this information using the powershell modules. This SDK doesn't appear to have followed this pattern.

I saw that TokenCredential has some support in version 12.14.0 but it read as having very limited support. https://learn.microsoft.com/en-us/dotnet/api/azure.storage.files.shares.shareclient.-ctor?view=azure-dotnet

The APIs I'm using were not listed here as supported by token credential.

[2023-06-12T16:18:47.336Z] Ensure quota failed for tenant ctus014128-016 - storage account: <redacted>
[2023-06-12T16:18:47.338Z] Azure.Storage.Files.Shares: This API does not support bearer tokens. For OAuth, use the Storage Resource Provider APIs instead. Learn more: https://aka.ms/azurefiles/restapi.
RequestId:26ebf81a-901a-00ba-4c49-9d3e8f000000
Time:2023-06-12T16:18:47.0785901Z
[2023-06-12T16:18:47.338Z] Status: 409 (This API does not support bearer tokens. For OAuth, use the Storage Resource Provider APIs instead. Learn more: https://aka.ms/azurefiles/restapi.)
[2023-06-12T16:18:47.338Z] ErrorCode: FileOAuthManagementApiRestrictedToSrp
[2023-06-12T16:18:47.339Z]
[2023-06-12T16:18:47.339Z] Content:
[2023-06-12T16:18:47.340Z] ?<?xml version="1.0" encoding="utf-8"?><Error><Code>FileOAuthManagementApiRestrictedToSrp</Code><Message>This API does not support bearer tokens. For OAuth, use the Storage Resource Provider APIs instead. Learn more: https://aka.ms/azurefiles/restapi.
RequestId:26ebf81a-901a-00ba-4c49-9d3e8f000000
Time:2023-06-12T16:18:47.0785901Z</Message></Error>
[2023-06-12T16:18:47.341Z]
[2023-06-12T16:18:47.341Z] Headers:
[2023-06-12T16:18:47.342Z] Server: Microsoft-HTTPAPI/2.0
[2023-06-12T16:18:47.342Z] x-ms-request-id: 26ebf81a-901a-00ba-4c49-9d3e8f000000
[2023-06-12T16:18:47.343Z] x-ms-client-request-id: ca2fa96c-fe42-4f70-9abf-25b2da77c75d
[2023-06-12T16:18:47.343Z] x-ms-error-code: FileOAuthManagementApiRestrictedToSrp
[2023-06-12T16:18:47.344Z] Date: Mon, 12 Jun 2023 16:18:46 GMT
[2023-06-12T16:18:47.344Z] Content-Length: 352
[2023-06-12T16:18:47.345Z] Content-Type: application/xml
[2023-06-12T16:18:47.345Z] .

I'm going to also look into this Azure Storage Resource Provider API surface to see if size and quota operations might be available through it using token auth without VNET peering.

[amnguye]

Yeah unfortunately looks like this is a limitation in the Files REST API (the SDK takes a dependency on the REST API, since it is used underneath).

But yes you would have to use the Azure Storage Resource Provider API instead. Which would be the Microsoft.Azure.Management.Storage instead.

@blueww Could you help out with the corresponding API they need to achieve setting the size and quota for a File Share (using Oauth).

[jstutson-rel]

I believe got it figured out with the move to the Azure Storage Resource Provider API. Thanks for confirming.

I do have a couple concerns with this move however.

1. I am curious if this API has the same or worse throttle limitations on calls per x seconds vs the APIs that ShareClient uses. We might be pushing near this limit.
2. The API spec mentions that ShareUsageBytes property returned by it is only "approximate" and may not include most recent writes, but it gives no additional detail as to how up-to-date this information is. Seconds? Minutes? Hours? It's crucial that I know how frequently this is updated.

I'll be testing both out but very much appreciate any answers here.

[amnguye]

@blueww Could you help answer this question?

[blueww]

@jstutson-rel To get File share from storage management plan SDK (the above PowerShell sample based on storage management plane API), Microsoft.Azure.Management.Storage is deprecated, please use Azure.ResourceManager.Storage instead. You can refer this sample for how to get File share usage.

For the 1st question on throttle limitations, you can refer this doc : https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#azure-storage-resource-provider-limits This should be read operation, and the limit is "800 per 5 minutes" per region.

For the 2nd question, this is a question on server API behavior, as client side tools/SDK owner, we might not be the best person to answer it. I would suggest you to raise a help request in Azure portal like following picture, then the related team will be engaged to help you? (If you have concern for the throttle limitation, you can also raise it in the help request.)

[jstutson-rel]

Thanks for confirming the request quota on Resource Manager but I'm curious how that compares to the quota on the ShareClient. Does anyone know?

In summary the optimal API for our file share monitoring has the following:

• A high request limit (800 per 5 minutes fits us for now but inhibits growth a bit)
• Up to the minute (or at absolute most 20 minute) accuracy of current usage bytes.
• OAuth/Token authentication with Azure AD (User Managed Identities)
• No VNet restriction

Looks like the resource manager version of these APIs will work well enough for us to monitor for adequate headroom and allocate more if needed from Azure Functions. The request limit remains a concern and I'm awaiting response from support on the SLO for usage stats, but I think we can close this.