Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.23k stars 4.58k forks source link

Unable to create ADF Azure Storage Linked Services #36099

Closed galvesribeiro closed 1 year ago

galvesribeiro commented 1 year ago

Library name and version

Azure.ResourceManager.DataFactory 1.0.0-beta.2

Query/Question

While trying to create an ADF resource with linked services, pipelines, etc, the creation of the Linked Service happens "successfully" however, the resource is not in a usable state.

Snippet:

private async Task<FactoryDatasetResource> CreateBlobDataset(
        DataFactoryResource adf,
        StorageBackupConfigEntry entry)
    {
        var prodResourceGroup = await this._azure.Production.GetResourceGroupAsync(entry.ResourceGroup);
        var storage = await prodResourceGroup.Value.GetStorageAccounts().GetAsync(entry.AccountName);

        StorageAccountKey key = default!;
        await foreach (var k in storage.Value.GetKeysAsync()) // Yeah this is ugly, but is what we have in the API...
        {
            key = k;
            break;
        }

        var blobLinkedService = new AzureBlobStorageLinkedService()
        {
            ConnectionString = BinaryData.FromObjectAsJson($"DefaultEndpointsProtocol=https;AccountName={entry.AccountName};AccountKey={key};EndpointSuffix=core.windows.net;"),
        };

        var data = new FactoryLinkedServiceData(blobLinkedService);

        var linkedService = await adf.GetFactoryLinkedServices().CreateOrUpdateAsync(WaitUntil.Completed, entry.AccountName, data);

        var linkedServiceReference = new FactoryLinkedServiceReference(FactoryLinkedServiceReferenceType.LinkedServiceReference, linkedService.Value.Data.Name);

        var blobDataSet = new BinaryDataset(linkedServiceReference);

        blobDataSet.Parameters.Add("container", new EntityParameterSpecification(EntityParameterType.String));

        blobDataSet.Parameters.Add("directory", new EntityParameterSpecification(EntityParameterType.String));

        blobDataSet.DataLocation = new AzureBlobStorageLocation()
        {
             // Couldn't find the proper class to represent the expression so, using anonymous type
            Container = BinaryData.FromObjectAsJson(new { value = "@dataset().container", type = "Expression" }),
            FolderPath = BinaryData.FromObjectAsJson(new { value = "@dataset().directory", type = "Expression" })
        };

        var ds = new FactoryDatasetData(blobDataSet);

        var blobDs = await adf.GetFactoryDatasets().CreateOrUpdateAsync(WaitUntil.Completed, entry.AccountName, ds);

        return blobDs.Value;
    }

When opening the Linked Service resource in the ADF designer, I see the resource there without any errors. But when I click on it and try to Test the connection (or if I try to run a pipeline that uses it) I get the following error:

Invalid storage connection string provided to 'UnknownLocation'. Check the storage connection string in configuration.
 No valid combination of account information found.
 Activity ID: e49844bf-4d63-4d76-a9fb-65a69127aa53.

I would appreciate any light on what may be missing here.

Thanks!

Environment

.NET SDK:
 Version:   7.0.102
 Commit:    4bbdd14480

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.3
 OS Platform: Darwin
 RID:         osx.13-arm64
 Base Path:   /usr/local/share/dotnet/sdk/7.0.102/

Host:
  Version:      7.0.2
  Architecture: arm64
  Commit:       d037e070eb

.NET SDKs installed:
  6.0.405 [/usr/local/share/dotnet/sdk]
  7.0.102 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.13 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
jsquire commented 1 year ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

ArthurMa1978 commented 1 year ago

@galvesribeiro Thank you for your feedback. Consider asking the service staff for confirmation as this can be a service behavior.

github-actions[bot] commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @shawnxzq @lmy269.

github-actions[bot] commented 1 year ago

Hi @galvesribeiro. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

galvesribeiro commented 1 year ago

I found the problem.

var blobLinkedService = new AzureBlobStorageLinkedService()
        {
            ConnectionString = BinaryData.FromObjectAsJson($"DefaultEndpointsProtocol=https;AccountName={entry.AccountName};AccountKey={key};EndpointSuffix=core.windows.net;"),
        };

This was using .ToString() while it should be using key.Value.

Sorry for the confusion and thanks for the replies.