search

microsoft / Spark-Hive-Tools

This is for issue/feedback tracking on Spark & Hive Tools
Creative Commons Attribution 4.0 International
6 stars 9 forks source link

Python path doesn't seem to be configurable #8

Open CalebFenton opened 4 years ago

CalebFenton commented 4 years ago

This extension tries to run "python" but that's python 2 on my system and I don't want to use that.

Everything else seems to respect python.pythonPath in my settings.json pointing to my virtual environment.

Would be nice if this respected the pythonPath or had it configurable somewhere. If it is configurable, it'd be nice if it were documented somewhere. If it's configurable and documented, maybe I'm just stupid and missed it?

mvanderlee commented 4 years ago

It's even worse than this. I uninstalled the plugin because of this, and it still uses the hdinsightJupyter. It's rendering VSCode useless for python dev.

zesluo commented 4 years ago

It has been fixed. Please try the latest version. Feel free to let me know if it still persist. You can click on the left bottom section of it to swith python. Thanks for filing the issue. image

teeupdata commented 4 years ago

I am using a conda virtual environment for a specific pyspark project with spark&hive extension enabled, and installed virtualenv using pip in this conda virtual env only. The correct python interpreter (the one under the conda virtual env) is picked up (confirmed by lower left corner), however, the message under output window still use python from conda base, where virtualenv is NOT installed:

2020-6-9:22:50:54] [Info] Exec python, with args: --version [2020-6-9:22:50:55] [Info] Python 3.7.4

[2020-6-9:22:50:55] [Info] Exec pip, with args: --version [2020-6-9:22:50:55] [Info] Link cluster ! [2020-6-9:22:50:57] [Info] pip 19.2.3 from .../opt/anaconda3/lib/python3.7/site-packages/pip (python 3.7) ---- this is the conda base env.

[2020-6-9:22:50:57] [Info] Exec virtualenv, with args: --version [2020-6-9:22:50:57] [Error] Error: spawn virtualenv ENOENT [2020-6-9:22:50:57] [Info] Exec python, with args: -m,virtualenv,--version [2020-6-9:22:50:57] [Info] .../opt/anaconda3/bin/python: No module named virtualenv --- this is the conda base env...

[2020-6-9:22:50:57] [Error] Exit with non zero 1

since the extension couldn't locate virtualenv, pyspark interactive mode is not working! How do I make pyspark interactive mode work with a conda virtual env?

zesluo commented 4 years ago

@teeupdata , thanks for your feedback. I will look into it.

zesluo commented 4 years ago

Are you the first time to install in tool?

teeupdata commented 4 years ago

first time trying pyspark interactive mode, have been using VS code developing Python locally

zesluo commented 4 years ago

you preinstalled a conda virtual environment?

teeupdata commented 4 years ago

yes, I created a conda virtual environment first, then config python interpreter for that workspace to use python executable under that conda virtual env. this is confirmed by lower left corner message.

zesluo commented 4 years ago

@teeupdata, thanks for your infor! May I reply to you later when I got a chance? Thanks a lot!

teeupdata commented 3 years ago

@zesluo I see this is closed. May I ask if this has been resolved? thanks.