ADLS gen 2 can be attached to an existing HDI cluster whose primary storage is NOT ADLS gen 2.

[natevictorlee]

• HDInisght - Storage options: Primary and Secondary

• Document link: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-compare-storage-options

  So, in the documentation, it says that the following combination for primary and secondary storage are not supported: General Purpose V1/General Purpose V2 (Primary Storage Option) + Data Lake Storage Gen2 (Secondary Storage) IS NOT SUPPORTED for HDInsight Version 3.6 + 4.0

  However, I tried this configuration by configuration the core-site.xml on Ambari:

  Using these two configurations pairs:

• fs.azure.account.auth.type.ADLSGEN2StorageName.dfs.core.windows.net : sharekey

• fs.azure.account.key.ADLSGEN2StorageName.dfs.core.windows.net : storageKey

  Configurations found on this site: https://iwasakims.github.io/trunk/hadoop-project/hadoop-azure/abfs.html#Default:_Shared_Key

  NOTE: Data Lake Storage Gen 2 is ENABLED on the Azure Portal for the storage (Secondary Storage Account)

So I configured the Data Lake Gen2 storage on core-site.xml along with my Gen Purpose V1 worked well and I can access ADLS Gen2 storage without issue.

Can someone tell if I am doing something wrong? If not, how come my additional configuration would make the two types of storage worked for HDInsight Cluster? Or the documentation mistakenly written that "General Purpose V1/General Purpose V2 (Primary Storage Option) + Data Lake Storage Gen2 (Secondary Storage) IS NOT SUPPORTED for HDInsight Version 3.6 + 4.0"?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: ef997d53-78df-9268-2514-7ec0878b2b25
• Version Independent ID: cd312664-8015-2b14-2933-c153a02f9fa5
• Content: Compare storage options for use with Azure HDInsight clusters
• Content Source: articles/hdinsight/hdinsight-hadoop-compare-storage-options.md
• Service: hdinsight
• GitHub Login: @hrasheed-msft
• Microsoft Alias: hrasheed

[KranthiPakala-MSFT]

@natevictorlee Thank you for reaching out. At this time we are reviewing the ask and will provide an update as appropriate.

[KranthiPakala-MSFT]

@natevictorlee Thanks for bringing this to us. Yes, it will work from an authentication aspect, but the engineering team have not run certification testing around this and there are a few known issues that might arise because of this configuration. Hence it is not recommended/supported.

Hope this helps.

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply and reopen the issue. We will gladly continue the discussion.

[l-rossetti]

I just want to highlight that the correct value is "SharedKey" and I'm able to correctly process data from such a secondary Data Lake 2 storage.