Masutani
commented
4 years ago I found this error occurs with Anaconda. When I switch to vanilla Python, the preparation scripts works. Most of errors I encountered after this were mismatch of environment between Anaconda and hdinsight local environment on this tools.
I finally succeeded to connect to EMR. There are some tips to work around some blocking matters.
- Connect to Azure subscription This is dummy connection to pass login check.
zesluo
I tryed connecting VSCode to EMR via this extension. I succeded to connect and select cluster on EMR, however it failed when launching script.
The message seems to be conflict on python environment between default hdinsight env. Are there any work around on this ?
[2020-7-17:21:47:18] [Info] Exec pip, with args: --version [2020-7-17:21:47:18] [Info] pip 20.0.2 from C:\Users****\anaconda3\lib\site-packages\pip (python 3.7) [2020-7-17:21:47:18] [Info] Exec virtualenv, with args: --version [2020-7-17:21:47:19] [Info] 16.7.5
[2020-7-17:21:47:19] [Info] Check Jupyter installation: [2020-7-17:21:47:19] [Info] Exec C:\Users*****.msvscode.hdinsight\hdinsightJupyter\Scripts\jupyter.exe, with args: --version [2020-7-17:21:47:19] [Info] Jupyter not installed [2020-7-17:21:47:19] [Info] Installing PySpark interactive virtual environment ...
[2020-7-17:21:47:19] [Info] Exec python, with args: -m,virtualenv,C:\Users*.msvscode.hdinsight\hdinsightJupyter ...[2020-7-17:21:47:21] [Info] Using base prefix 'C:\Users\***\anaconda3' No LICENSE.txt / LICENSE found in source New python executable in C:\Users\.msvscode.hdinsight\hdinsightJupyter\Scripts\python.exe copying C:\Users*\anaconda3\python.exe => C:\Users*.msvscode.hdinsight\hdinsightJupyter\Scripts\python.exe Installing setuptools, pip, wheel...
.............