Closed furmangg closed 6 years ago
Thanks for the PR. I don't think, removing fs.adl.impl and fs.AbstractFileSystem.adl.impl would work. Did you create the cluster without ADLS support and then added configurations ? Or you had created the cluster with ADLS to start with ?
Yes. When you create a 3.6 cluster with Data Lake Store as the default storage those two properties are already there. Leave them alone. Don't delete them or change them.
I don't know the behavior if Data Lake Store is secondary storage.
Those settings also have a slightly different value on a 3.6 cluster also. But they were there so I skipped them. Hope that's clear.
The guide is meant for adding the ADLS storage with presto even when the cluster was not initially not created with ADLS support. So, in that case, you need to add the properties. You can say that if they are already there, we dont need to change it. Can you make those changes ?
I made an edit to the README. I validated that even if it's not the default storage account my instructions are right. And I made sure all 6 properties were listed for HDI3.6. And I fixed the values you should use for 3.6. Let me know if that's not quite right yet.
Thanks. I am OOF till next week. I will look at this once I am back. Thanks for the patience !
Do my edits look good?
@dharmeshkakadia, it has been a while but I wanted to check if you have been able to review my edits.
Sorry, this slipped out during the holidays.
Thanks for the contribution and patience.
Please consider keeping this documentation updated to work with HDI 3.6. I haven't tried on 3.5, but the first two properties are already set and have slightly different values on a HDI 3.6 cluster on ADLS. If I skip the first two properties (fs.adl.impl and fs.AbstractFileSystem.adl.impl) on 3.6 it works.
Also mentioning it works in an ARM template is important as many people spin up and spin down clusters and aren't able to manually touch the config every time.
We also need to get this documentation into the https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-install-presto as it currently says "It must use Azure Storage as the data store. Using Presto on a cluster that uses Azure Data Lake Store as the storage option is not yet supported." Are you able to do that or do I need to create a pull request on the Azure docs too?