modules in environment not visible

[erwindassen]

Hi there,

I'm having an issue with the following: when I activate a conda environment with xonda the modules in that environment are not visible. For example, if I have numpy in the mlenv-py36 environment but not in the root environment:

erwindassen@kerbin:~$ xonsh 
erwindassen@kerbin ~ $ conda activate mlenv-py36
(mlenv-py36) erwindassen@kerbin ~ $ import numpy
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
ImportError: No module named 'numpy'

The modules of the root environment are visible though. This has to do with the PYTHONPATH not being updated:

(mlenv-py36) erwindassen@kerbin ~ $ import sys
(mlenv-py36) erwindassen@kerbin ~ $ sys.path
['/usr/local/share/miniconda3/bin',
 '/usr/local/share/miniconda3/lib/python35.zip',
 '/usr/local/share/miniconda3/lib/python3.5',
 '/usr/local/share/miniconda3/lib/python3.5/plat-linux',
 '/usr/local/share/miniconda3/lib/python3.5/lib-dynload',
 '/usr/local/share/miniconda3/lib/python3.5/site-packages']

Note that this is the root environment's PYTHONPATH.

As expected, if inside the mlenv-py36 environment I run python, inside its shell, I can import numpy since the paths are then updated.

I think it would be great if when xonda activating an environment the sys.path would be updated. I can do it myseld on my local machine but I would hard code it since I don't know how to obtain the necessary paths from conda. Do you have any idea?

[gforsyth]

Hey @erwindassen -- thanks for opening this. There have been some discussions about this and how we might accomplish it without breaking everything. If we arbitrarily replace things in sys.path, xonsh will get very unhappy with us.

ping @astronouth7303 who I think had mentioned this re: vox

[AstraLuma]

(Matching vox issue: xonsh/xontrib-vox#5)

I don't think vox itself will be unhappy (the odds of collision should be low, and everything it needs should be loaded), but there is the problem of what happens if you do this a few times in a session? Since there's a pretty high chance not all references will get cleaned up, something of the old module will remain even if it's "unloaded" (removed from sys.modules).

There is the risk of breaking xonsh itself, though. In general, xonsh itself is not strongly isolated from the prompt, so if there is a collision or bad global state or some other problem, it's possible to break your shell and require closing it.

These issues are why I suggested starting a subshell in the vox version, but then you can't (easily) pass data between them.

Opening a (I)Python repl should work just fine, though.

[erwindassen]

Indeed opening a new REPL works but imho this defeats one of the great things about xonsh.

That aside, I understand the cautionary words. What I was thinking is that the necessary path changes are very deterministic. I mean, it is a conda thing that needs a few dirs. Perhaps one could do some sanity checks to compare them to the current environment and substitute it if necessary? I don't know... just throwing ideas here. Does anyone have a reference (if it exists) where conda tells us how to get these dirs programatically? I would love to spend some more time on this but in the coming few weeks it is impossible.

EDIT: I assume xonsh, xonda will have to be installed in the environments you want to be able to switch to.

[gforsyth]

on *nix, there's a .conda/environments.txt file that lists the locations of all of the environments. When you create an environment (or possible when it's activated for the first time?) conda symlinks itself into the appropriate bin/ directory, so you are always running the "base" conda. There's also an envvar $CONDA_DEFAULT_ENV that sets which environment conda installs or removes packages from.

As for the sys.path stuff, I don't think conda is actually handling those, just relying on Python's machinery for that. Python will, by default, look for the lib directory that's on the same level as the bin directory. Then it will tack on anything specified by any pth files that it finds.

So you could do that yourself, really. Especially for the main lib dir. The pth stuff might be a little trickier but still quite doable.

[AstraLuma]

See the vox bug. A lot of the machinery can be handled for us (if you treat environments as sites), but you have to do some book keeping so that state can be restored when the user's done.

[gforsyth]

huh... I missed that the first time around. site.addsitedir seems pretty solid.

We could always do this in a more primitive way and explicitly not support people hacking their sys.path in session. Then it seems we wouldn't have to do much bookkeeping, just pop everything out of sys.path, then do an addsitedir for whichever the current site-packages directory is, no?

[gforsyth]

I'll add that I just tried this by hand and was able to do the sort of vox enter you were proposing with one of my environments without apparent side-effects. Obviously needs more testing but it seems to work ok

[AstraLuma]

I figured saving sys.path and sys.modules.keys() would be the simple, robust way to be able to restore things.

What clean up did you test with?

Be sure to test how it handles it when objects stick around, and if you have multiple environments with conflicting packages. (ie, enter env1, leave, and then enter env2.)

[erwindassen]

I thinks this approach is the right one. Gonna try here as well. Thank you!

[erwindassen]

Oh, but you have to be sure to have xonsh/xonda and python with the same versions inside the environment (if you pop everything from sys.path). There's a whole dependency nightmare hidden here. For example, I added the site-package dir from the environment with site.addsitedir and tried to import pandas this failed because the compiled dependencies (dynlibs) where not found.

Perhaps the best is to have xonda activate to actually spawn a subprocess in the right environment? In this way the only requirement would be for xonsh to be installed in the desired environment.

[gforsyth]

"It turns out shells have some tricky edge cases."

Yeah, this is always going to be a little ugly. The dynlibs live (I think) in lib/python3.x/lib-dynload which you would need to add separately with addsitedir.

Also note that if you import a library in one environment, import caching will mean that you can't reimport it from a separate environment.

[erwindassen]

I was changing my xonsh settings and came across this:

$UPDATE_OS_ENVIRON
If True ``os_environ`` will always be updated when the xonsh
environment changes. The environment can be reset to the default
value by calling ``__xonsh_env__.undo_replace_env()``
default value: False
current value: False

Is this perhaps relevant to our issue? I've set it to true but don't see any effect.

[gforsyth]

Hey @erwindassen -- no, I don't think that will have an impact on us. That was added a while back to ensure parity between the xonsh envvars and those reported by os.environ, but that shouldn't impact the sys.path in any meaningful way.