search

dharmeshkakadia / presto-hdinsight

Presto on Azure HDInsight
8 stars 24 forks source link

Improving instructions for HDI 3.6 #17

Closed furmangg closed 6 years ago

furmangg commented 7 years ago

Please consider keeping this documentation updated to work with HDI 3.6. I haven't tried on 3.5, but the first two properties are already set and have slightly different values on a HDI 3.6 cluster on ADLS. If I skip the first two properties (fs.adl.impl and fs.AbstractFileSystem.adl.impl) on 3.6 it works.

Also mentioning it works in an ARM template is important as many people spin up and spin down clusters and aren't able to manually touch the config every time.

We also need to get this documentation into the https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-install-presto as it currently says "It must use Azure Storage as the data store. Using Presto on a cluster that uses Azure Data Lake Store as the storage option is not yet supported." Are you able to do that or do I need to create a pull request on the Azure docs too?

dharmeshkakadia commented 7 years ago

Thanks for the PR. I don't think, removing fs.adl.impl and fs.AbstractFileSystem.adl.impl would work. Did you create the cluster without ADLS support and then added configurations ? Or you had created the cluster with ADLS to start with ?

furmangg commented 7 years ago

Yes. When you create a 3.6 cluster with Data Lake Store as the default storage those two properties are already there. Leave them alone. Don't delete them or change them.

I don't know the behavior if Data Lake Store is secondary storage.

furmangg commented 7 years ago

Those settings also have a slightly different value on a 3.6 cluster also. But they were there so I skipped them. Hope that's clear.

dharmeshkakadia commented 6 years ago

The guide is meant for adding the ADLS storage with presto even when the cluster was not initially not created with ADLS support. So, in that case, you need to add the properties. You can say that if they are already there, we dont need to change it. Can you make those changes ?

furmangg commented 6 years ago

I made an edit to the README. I validated that even if it's not the default storage account my instructions are right. And I made sure all 6 properties were listed for HDI3.6. And I fixed the values you should use for 3.6. Let me know if that's not quite right yet.

dharmeshkakadia commented 6 years ago

Thanks. I am OOF till next week. I will look at this once I am back. Thanks for the patience !

furmangg commented 6 years ago

Do my edits look good?

furmangg commented 6 years ago

@dharmeshkakadia, it has been a while but I wanted to check if you have been able to review my edits.

dharmeshkakadia commented 6 years ago

Sorry, this slipped out during the holidays.

dharmeshkakadia commented 6 years ago

Thanks for the contribution and patience.