gforsyth / xonda

DEPRECATED: (previously a thin wrapper around conda for xonsh)
BSD 3-Clause "New" or "Revised" License
11 stars 9 forks source link

modules in environment not visible #7

Closed erwindassen closed 5 years ago

erwindassen commented 7 years ago

Hi there,

I'm having an issue with the following: when I activate a conda environment with xonda the modules in that environment are not visible. For example, if I have numpy in the mlenv-py36 environment but not in the root environment:

erwindassen@kerbin:~$ xonsh 
erwindassen@kerbin ~ $ conda activate mlenv-py36
(mlenv-py36) erwindassen@kerbin ~ $ import numpy
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
ImportError: No module named 'numpy'

The modules of the root environment are visible though. This has to do with the PYTHONPATH not being updated:

(mlenv-py36) erwindassen@kerbin ~ $ import sys
(mlenv-py36) erwindassen@kerbin ~ $ sys.path
['/usr/local/share/miniconda3/bin',
 '/usr/local/share/miniconda3/lib/python35.zip',
 '/usr/local/share/miniconda3/lib/python3.5',
 '/usr/local/share/miniconda3/lib/python3.5/plat-linux',
 '/usr/local/share/miniconda3/lib/python3.5/lib-dynload',
 '/usr/local/share/miniconda3/lib/python3.5/site-packages']

Note that this is the root environment's PYTHONPATH.

As expected, if inside the mlenv-py36 environment I run python, inside its shell, I can import numpy since the paths are then updated.

I think it would be great if when xonda activating an environment the sys.path would be updated. I can do it myseld on my local machine but I would hard code it since I don't know how to obtain the necessary paths from conda. Do you have any idea?

gforsyth commented 7 years ago

Hey @erwindassen -- thanks for opening this. There have been some discussions about this and how we might accomplish it without breaking everything. If we arbitrarily replace things in sys.path, xonsh will get very unhappy with us.

ping @astronouth7303 who I think had mentioned this re: vox

AstraLuma commented 7 years ago

(Matching vox issue: xonsh/xontrib-vox#5)

I don't think vox itself will be unhappy (the odds of collision should be low, and everything it needs should be loaded), but there is the problem of what happens if you do this a few times in a session? Since there's a pretty high chance not all references will get cleaned up, something of the old module will remain even if it's "unloaded" (removed from sys.modules).

There is the risk of breaking xonsh itself, though. In general, xonsh itself is not strongly isolated from the prompt, so if there is a collision or bad global state or some other problem, it's possible to break your shell and require closing it.

These issues are why I suggested starting a subshell in the vox version, but then you can't (easily) pass data between them.

Opening a (I)Python repl should work just fine, though.

erwindassen commented 7 years ago

Indeed opening a new REPL works but imho this defeats one of the great things about xonsh.

That aside, I understand the cautionary words. What I was thinking is that the necessary path changes are very deterministic. I mean, it is a conda thing that needs a few dirs. Perhaps one could do some sanity checks to compare them to the current environment and substitute it if necessary? I don't know... just throwing ideas here. Does anyone have a reference (if it exists) where conda tells us how to get these dirs programatically? I would love to spend some more time on this but in the coming few weeks it is impossible.

EDIT: I assume xonsh, xonda will have to be installed in the environments you want to be able to switch to.

gforsyth commented 7 years ago

on *nix, there's a .conda/environments.txt file that lists the locations of all of the environments. When you create an environment (or possible when it's activated for the first time?) conda symlinks itself into the appropriate bin/ directory, so you are always running the "base" conda. There's also an envvar $CONDA_DEFAULT_ENV that sets which environment conda installs or removes packages from.

As for the sys.path stuff, I don't think conda is actually handling those, just relying on Python's machinery for that. Python will, by default, look for the lib directory that's on the same level as the bin directory. Then it will tack on anything specified by any pth files that it finds.

So you could do that yourself, really. Especially for the main lib dir. The pth stuff might be a little trickier but still quite doable.

AstraLuma commented 7 years ago

See the vox bug. A lot of the machinery can be handled for us (if you treat environments as sites), but you have to do some book keeping so that state can be restored when the user's done.

gforsyth commented 7 years ago

huh... I missed that the first time around. site.addsitedir seems pretty solid.

We could always do this in a more primitive way and explicitly not support people hacking their sys.path in session. Then it seems we wouldn't have to do much bookkeeping, just pop everything out of sys.path, then do an addsitedir for whichever the current site-packages directory is, no?

gforsyth commented 7 years ago

I'll add that I just tried this by hand and was able to do the sort of vox enter you were proposing with one of my environments without apparent side-effects. Obviously needs more testing but it seems to work ok

AstraLuma commented 7 years ago

I figured saving sys.path and sys.modules.keys() would be the simple, robust way to be able to restore things.

What clean up did you test with?

Be sure to test how it handles it when objects stick around, and if you have multiple environments with conflicting packages. (ie, enter env1, leave, and then enter env2.)

erwindassen commented 7 years ago

I thinks this approach is the right one. Gonna try here as well. Thank you!

erwindassen commented 7 years ago

Oh, but you have to be sure to have xonsh/xonda and python with the same versions inside the environment (if you pop everything from sys.path). There's a whole dependency nightmare hidden here. For example, I added the site-package dir from the environment with site.addsitedir and tried to import pandas this failed because the compiled dependencies (dynlibs) where not found.

Perhaps the best is to have xonda activate to actually spawn a subprocess in the right environment? In this way the only requirement would be for xonsh to be installed in the desired environment.

gforsyth commented 7 years ago

"It turns out shells have some tricky edge cases."

Yeah, this is always going to be a little ugly. The dynlibs live (I think) in lib/python3.x/lib-dynload which you would need to add separately with addsitedir.

Also note that if you import a library in one environment, import caching will mean that you can't reimport it from a separate environment.

erwindassen commented 7 years ago

I was changing my xonsh settings and came across this:

$UPDATE_OS_ENVIRON
If True ``os_environ`` will always be updated when the xonsh
environment changes. The environment can be reset to the default
value by calling ``__xonsh_env__.undo_replace_env()``
default value: False
current value: False

Is this perhaps relevant to our issue? I've set it to true but don't see any effect.

gforsyth commented 7 years ago

Hey @erwindassen -- no, I don't think that will have an impact on us. That was added a while back to ensure parity between the xonsh envvars and those reported by os.environ, but that shouldn't impact the sys.path in any meaningful way.