Jupyter path not created correctly with conda clone

[cbeebe44]

Hi, I have the identical issue open at conda/conda.
I need to know how to change the jupyter executable, config, and data paths used in an instance of jupyter notebook/console started from within a conda environment.
I've used conda to clone an environment. When initiating jupyter notebook/qtconsole from the cloned environment I see that the python bin, config, and data files are from the source environment I cloned from.
jupyter --paths is identical in both environments.
from within jupyter apps, os.environ, jupyter_core.paths.jupyter_config_path(), and sys.executable are all identical.
If starting instance of python within environment, os.environ and sys.executable behave as expected and reflect the current conda environment directories respectively.
I suspect the mixup is due to conda create --clone

[cbeebe44]

please help

[kevin-bates]

I'm not able to reproduce this. I'm wondering if the top line of your respective <path-to-conda-env>/bin/jupyter-xxx scripts are referencing the python from the cloned env, rather than the new env. However, when I clone my env, all scripts are updated to reference the correct env-relative python. I'm using conda 4.7.11, but notice you're using 4.8.1 and I doubt a regression of this magnitude would exist.

At any rate, could you please check the top line of jupyter scripts to ensure they're referencing the correct python?

That's about the only idea I have (sorry). Hopefully others, with much deeper python/conda experience than myself, have some ideas.

[cbeebe44]

I'm not sure if I'm understanding this correctly but all the jupyter scripts are executable files (.exe) located in ~\miniconda\envs\myenv\Scripts\? If so I don't know how to check the first line. If opened in text reader it is all unreadable.

[cbeebe44]

steps to reproduce:

conda create -n env1
conda activate env1
pip install jupyter
jupyter --paths
conda create --clone env1 -n env2
conda activate env2
jupyter --paths

the paths of both environments point to same directory.

[kevin-bates]

Hmm - just goes to show my naïveté on Windows wrt Jupyter.

Perhaps Windows is a different beast in this respect and unix shell-style scripts located on unix systems in, for example, /users/aka/miniconda3/envs/env1/bin are actually executable files located in c:\users\aka\miniconda\envs\env1\Scripts on Windows platforms?

If that's the case, then I suspect conda is unable to perform the "python-path-massaging" it does to unix script files on Windows executables during clone operations. Seems like you'd need to perform a conda update relative to the appropriate jupyter packages for the new env so that the executable is updated for the new env.

I'm sure there are Jupyter veterans that have run into this.

[jasongrout]

the paths of both environments point to same directory.

What is that path? Is it inside the first environment?

[cbeebe44]

From the second environment (cloned one) jupyter is pointing to the first environment (source one).

[jasongrout]

This works for me:

conda create -n env1 pip
conda activate env1
pip install jupyter
jupyter --paths
conda create --clone env1 -n env2
conda activate env2
jupyter --paths

The second jupyter --paths correctly has the sys-prefix level pointing to env2, not env1.

A couple of thoughts:

1. Are you using the latest conda?
2. What if you install jupyter using conda instead of pip? conda create -n env1 -c conda-forge jupyter
3. If none of those sheds light on the issue, perhaps this would be better to take up on the conda issue tracker, since the most interesting step here is the conda clone.

[cbeebe44]

@kevin-bates the scripts in c:\users\aka\miniconda\envs\env1\scripts\ and c:\users\aka\miniconda\envs\env2\scripts\ are .exe files. I think they were installed this way by pip. I performed a conda update jupyter from the new environment which gave me an error because conda does not recognize pip installed packages. So I did a conda install jupyter which installed jupyter packages and added .py files for jupyter scripts in the scripts folder. Now jupyter apps are not working. This has happened before with using conda to install jupyter packages on my python 3.5 environment, that is why I used pip which worked out.

[jasongrout]

So I did a conda install jupyter which installed jupyter packages and added .py files for jupyter scripts in the scripts folder.

I think conda installing over a previously pip-installed package is probably not supported. If you conda install, I would do it in a new clean environment.

[cbeebe44]

A couple of thoughts:

1. Are you using the latest conda?

2. What if you install jupyter using conda instead of pip? `conda create -n env1 -c conda-forge jupyter`

3. If none of those sheds light on the issue, perhaps this would be better to take up on the conda issue tracker, since the most interesting step here is the conda clone.

@jasongrout

1. Yes; conda 4.8.1
2. Installing jupyter with conda has not been working with python 3.5 enviroonment
3. I have an open issue and conda/conda. I've taken it up with conda-forge but they do not support python 3.5 environments.

[cbeebe44]

@jasongrout , yes. I think a clean environment is where I will start again. I was just hoping there was a way to change this which might be useful to know in the future.

[cbeebe44]

with a conda install of jupyter in python 3.5 environment I get this error when starting jupyter console:

(test02)PS C:\Users\Aka> jupyter console
[ZMQTerminalIPythonApp] ERROR | Failed to run command:
['C:/Users/Aka/Miniconda3/envs/test02/bin/python', '-m', 'ipykernel_launcher', '-f', 'C:\\Users\\Aka\\AppData\\Roaming\\jupyter\\runtime\\kernel-2404.json']
    PATH='C:\\Users\\Aka\\Miniconda3\\envs\\test02\\Library\\bin;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\Library\\bin;C:\\Users\\Aka\\Miniconda3\\envs\\test02;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\Library\\mingw-w64\\bin;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\Library\\usr\\bin;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\Library\\bin;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\Scripts;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\bin;C:\\Users\\Aka\\Miniconda3\\condabin;C:\\Program Files (x86)\\Common Files\\Oracle\\Java\\javapath;C:\\Program Files\\Java\\jdk-13.0.2\\bin;C:\\Rtools\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;C:\\Program Files (x86)\\PuTTY;C:\\Program Files\\Git\\cmd;C:\\Program Files\\Microsoft SQL Server\\130\\Tools\\Binn;C:\\Program Files\\MATLAB\\R2017a\\runtime\\win64;C:\\Program Files\\MATLAB\\R2017a\\bin;C:\\Program Files\\Microsoft\\Web Platform Installer;C:\\WINDOWS\\System32\\OpenSSH;C:\\Strawberry\\c\\bin;C:\\Strawberry\\perl\\site\\bin;C:\\Strawberry\\perl\\bin;C:\\Program Files\\Intel\\WiFi\\bin;C:\\Program Files\\Common Files\\Intel\\WirelessCommon;C:\\Program Files (x86)\\Intel\\Intel(R) Management Engine Components\\DAL;C:\\Program Files\\Intel\\Intel(R) Management Engine Components\\DAL;C:\\Program Files\\Mu\\bin;C:\\Program Files (x86)\\Microsoft VS Code\\bin;C:\\Program Files (x86)\\nodejs;C:\\Program Files\\geckodriver-v0.26.0-win64;C:\\Program Files (x86)\\Microsoft VS Code\\bin;C:\\Users\\Aka\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\Aka\\AppData\\Local\\GitHubDesktop\\bin;C:\\Users\\Aka\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Program Files\\Intel\\WiFi\\bin;C:\\Program Files\\Common Files\\Intel\\WirelessCommon;C:\\Users\\Aka\\AppData\\Roaming\\npm;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\lib\\site-packages\\pywin32_system32;C:\\Users\\Aka\\Miniconda3\\envs\\test02\\lib\\site-packages\\pywin32_system32'
    with kwargs:
{'stderr': None, 'stdin': -1, 'stdout': None, 'cwd': None, 'close_fds': False}

Traceback (most recent call last):
  File "C:\Users\Aka\Miniconda3\envs\test02\Scripts\jupyter-console-script.py", line 9, in <module>
    sys.exit(main())
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_core\application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\traitlets\config\application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\decorator.py:decorator-gen-117>", line 2, in initialize
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_console\app.py", line 142, in initialize
    self.init_shell()
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_console\app.py", line 109, in init_shell
    JupyterConsoleApp.initialize(self)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_client\consoleapp.py", line 334, in initialize
    self.init_kernel_manager()
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_client\consoleapp.py", line 288, in init_kernel_manager
    self.kernel_manager.start_kernel(**kwargs)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_client\manager.py", line 259, in start_kernel
    **kw)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_client\manager.py", line 204, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\site-packages\jupyter_client\launcher.py", line 138, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\Users\Aka\Miniconda3\envs\test02\lib\subprocess.py", line 957, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

note: this is a clean environment that I just created.

conda create -n test02 python=3.5 jupyter
conda activate test02
conda install pywin32 

[jasongrout]

I'm not on Windows, but FYI, this works for me:

conda create -n test02 -c conda-forge python=3.5 jupyter
conda activate test02
jupyter console

Perhaps someone who has Windows can test your instructions on Windows.

FYI, I have this package list:

# Name                    Version                   Build  Channel
appnope                   0.1.0                    py35_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
bzip2                     1.0.8                h0b31af3_2    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2018.8.24             py35_1001    conda-forge
decorator                 4.4.1                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
entrypoints               0.2.3                    py35_2    conda-forge
icu                       58.2              h0a44026_1000    conda-forge
ipykernel                 5.1.0              pyh24bf2e0_0    conda-forge
ipython                   7.0.1            py35h24bf2e0_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.4.2                      py_0    conda-forge
jedi                      0.12.1                   py35_0    conda-forge
jinja2                    2.10.3                     py_0    conda-forge
jpeg                      9c                h1de35cc_1001    conda-forge
jsonschema                2.6.0                    py35_2    conda-forge
jupyter                   1.0.0                      py_2    conda-forge
jupyter_client            5.3.3                      py_0    conda-forge
jupyter_console           6.0.0                      py_0    conda-forge
jupyter_core              4.5.0                      py_0    conda-forge
libcxx                    9.0.1                         1    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libpng                    1.6.37               h2573ce8_0    conda-forge
libsodium                 1.0.18               h01d97ff_0    conda-forge
markupsafe                1.0              py35h470a237_1    conda-forge
mistune                   0.8.3            py35h470a237_2    conda-forge
nbconvert                 5.6.0                      py_0    conda-forge
nbformat                  5.0.4                      py_0    conda-forge
ncurses                   6.1               h0a44026_1002    conda-forge
notebook                  5.7.0                    py35_0    conda-forge
openssl                   1.0.2u               h0b31af3_0    conda-forge
pandoc                    2.9.1.1                       0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.5.2                      py_0    conda-forge
pexpect                   4.6.0                    py35_0    conda-forge
pickleshare               0.7.5                    py35_0    conda-forge
pip                       18.0                  py35_1001    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.10                     py_0    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pygments                  2.5.2                      py_0    conda-forge
pyqt                      5.6.0            py35h8210e8a_7    conda-forge
python                    3.5.5                h5001a0f_2    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pyzmq                     17.1.2           py35hae99301_0    conda-forge
qt                        5.6.2             h822fa55_1013    conda-forge
qtconsole                 4.6.0                      py_0    conda-forge
readline                  7.0               hcfe32e1_1001    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                40.4.3                   py35_0    conda-forge
simplegeneric             0.8.1                      py_1    conda-forge
sip                       4.18.1           py35hfc679d8_0    conda-forge
six                       1.11.0                   py35_1    conda-forge
sqlite                    3.28.0               h9721f7c_0    conda-forge
terminado                 0.8.1                    py35_1    conda-forge
testpath                  0.4.4                      py_0    conda-forge
tk                        8.6.10               hbbe82c9_0    conda-forge
tornado                   5.1.1            py35h470a237_0    conda-forge
traitlets                 4.3.2                    py35_0    conda-forge
wcwidth                   0.1.8                      py_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.32.0                py35_1000    conda-forge
widgetsnbextension        3.4.2                    py35_0    conda-forge
xz                        5.2.4             h1de35cc_1001    conda-forge
zeromq                    4.2.5                hfc679d8_4    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge

[ivanov]

Hey there, I'm going through old issues and it seems to me that it makes sense to close this one. Hope you figured out a way to get a working environment, @cbeebe44 !

Thanks everyone and happy hacking! :bowtie: