Closed DonJayamanne closed 2 years ago
@jvaesteves Thanks for filing this and I'm sorry its not working as expected. Please could you:
Jupyter
output panel, you can get to the output panel using the command Jupyter: Show Output
(please copy all of the output and paste it here)$SPARKMAGIC_LOCATION
, asking so that I can map this to the values I see in the logs.SPARKMAGIC_LOCATION
points to ~/Library/Caches/pypoetry/virtualenvs/jupyter-playground-FOTz9V3J-py3.10/lib/python3.10/site-packages
I was also able to enable two of the kernels on VSCode list, but the PySpark kernel (that actually uses Python) still does not appear as an option. Also, the list shows these kernels pointing to a path that does not exists on my filesystem and does not correspond to jupyter kernelspec list
.
Available kernels:
pysparkkernel /Users/myuser/Library/Jupyter/kernels/pysparkkernel
sparkkernel /Users/myuser/Library/Jupyter/kernels/sparkkernel
sparkrkernel /Users/myuser/Library/Jupyter/kernels/sparkrkernel
python3 /Users/myuser/Library/Caches/pypoetry/virtualenvs/jupyter-playground-FOTz9V3J-py3.10/share/jupyter/kernels/python3
Furthermore, I can select and use PySpark as kernel on the notebook web interface on localhost.
@jvaesteves , thanks for the logs.
Also, the list shows these kernels pointing to a path that does not exists on my filesystem and does not correspond to
Which one are you referring to from the above UI.
but the PySpark kernel (that actually uses Python) still does no
Which one are you referring to form the list of kernels.
Available kernels:
Please could you open the kernelspec.json file in the following directories and dump the contents here so I can see this.
pysparkkernel /Users/myuser/Library/Jupyter/kernels/pysparkkernel
python3 /Users/myuser/Library/Caches/pypoetry/virtualenvs/jupyter-playground-FOTz9V3J-py3.10/share/jupyter/kernels/python3
Which one are you referring to from the above UI.
The ones on the Jupyter Kernel sections that points to /python
Which one are you referring to form the list of kernels.
pysparkkernel
Please could you open the kernelspec.json file in the following directories and dump the contents here so I can see this.
There you go. python3 | pysparkkernel
@jvaesteves The problem is the kernlespec.json files do not contain the fully qualified paths to python. If you check the contents (see below):
{
"argv": [
"python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3 (ipykernel)",
"language": "python",
"metadata": {
"debugger": true
}
}
The first value in argv
is just python
, hence the value doesn't contian the fully qualified path to pyhton.
If you were to launch Jupyter in your terminal from another Python environment, say you launched jupyter from /usr/bin/python3
, then you'd run into similar issues.
To get around this, i'd suggest finding the fully qualified path to the Python environment and then updating the above kernlspec with that. as follows:
{
"argv": [
"/usr/bin/python3", // or change to `/Users/myuser/Library/Caches/pypoetry/virtualenvs/jupyter-playground-FOTz9V3J-py3.10/bin/python`
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3 (ipykernel)",
"language": "python",
"metadata": {
"debugger": true
}
}```
let me know if that works for you.
@DonJayamanne Thanks a lot for the help. With this change, now the PySpark works correctly! :)
@jvaesteves Thanks for getting back with the confirmation, its not ideal but I'm glad you got things working.
I'd like to ensure we fix this workflow, hence have a few questions:
which python
point to in your terminal where you launch Jupyter for PySparkPython: Select Interpreter
, The above information would help us get things right so you don't have to (hopefully) do this again.
- What is the environment in which you have spark installed, is that Poetry?
Yes
How do you launch Jupyter in your terminal?
What is your active python environment in the Python extension
When I select the notebook file on my project, Jupyter starts and loads the PySpark kernel by itself. It is not needed to start Poetry venv, and even when I select the Interpreter on VSCode to be any other than from what is used by Poetry, the kernel bypasses it and uses the venv one (confirmed with the which python
on the notebook).
thank you very much, few more questions, thanks for your support and patience
When I select the notebook file on my project, Jupyter starts and loads the
how do you do this in the terminal. please could you paste the commands here
also part could you let me know what you get when you run the following two commands
which python which jupyter
Hello @DonJayamanne , sorry for taking so long to reply to you.
how do you do this in the terminal. please could you paste the commands here
That's the point, I don't execute any other command on terminal besides the ones I listed on the Steps to reproduce
section of the issue. The only other command that is executed automatically when I open my terminal on VSCode is to activate my venv.
also part could you let me know what you get when you run the following two commands
which python
/Users/myuser/Library/Caches/pypoetry/virtualenvs/jupyter-playground-FOTz9V3J-py3.10/bin/python
which jupyter
/Users/myuser/Library/Caches/pypoetry/virtualenvs/jupyter-playground-FOTz9V3J-py3.10/bin/jupyter
@jvaesteves From what I understand things are working now, after you updated the kernelspec.json file, is that right?
Closing this issue as its been over 4 weeks, since the information was requested. We'll be happy to reopen the issue when the requested information has been provided.
Discussed in https://github.com/microsoft/vscode-jupyter/discussions/11125