search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.26k stars 21.43k forks source link

Cannot get livy to work with local jar on head node #29933

Closed jdecowski18 closed 5 years ago

jdecowski18 commented 5 years ago

Hi, I followed this section to start a jar deployed to the headnodes: Updates to Livy configuration starting with HDInsight 3.5 version.

I'm using HDI 3.6 and no default-livy config exists in the ambari dashboard. I tried adding livy.file.local-dir-whitelist to the custom-livy config and livy cannot handle the request when I give it a local file path (body below). { "file":"/home/livy/SparkSimpleApp.jar", "className":"com.microsoft.spark.example.WasbIOTest" }

Can this section of the documentation be enhanced?

Thanks!

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

CHEEKATLAPRADEEP-MSFT-zz commented 5 years ago

@jdecowski18 Thanks for bringing this to our attention. I have assigned the issue to the content author to review further and update the document as appropriate.

JasonWHowell commented 5 years ago

@jdecowski18 We are a little behind, and couldn't take a look yet. Did you have any breakthroughs to get this scenario to work in HDI 3.6? Thanks, Jason

jdecowski18 commented 5 years ago

Thanks for the response Jason. No major breakthroughs for me yet.

hrasheed-msft commented 5 years ago

@jdecowski18 Given that the local jar option was disabled by default since 3.5, the feedback from the product group is that most likely the local-jar option has been disabled for security reasons and the editing of the livy config is not supported or encouraged by them for similar reasons. Alternatively, you can try using the azure storage option for hosting the jar as mentioned in the doc. We will update the document to reflect this guidance.

hrasheed-msft commented 5 years ago

@CHEEKATLAPRADEEP-MSFT we have updated the docs to remove the steps causing the confusion and just left the guidance to use wasb file paths. #please-close

CHEEKATLAPRADEEP-MSFT-zz commented 5 years ago

@jdecowski18 We will now proceed to close this thread. If there are further questions regarding this matter, please comment and we will gladly continue the discussion.