env base not activated on Windows

[alkorang]

Please see https://github.com/JuliaPy/PyCall.jl/issues/561.

The point is executing just Python binary does not activate base env. We need to explicitly activate base env on Windows.

C:\Users\alkorang>.julia\packages\Conda\m7vem\deps\usr\python
Python 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "C:\Users\alkorang\.julia\packages\Conda\m7vem\deps\usr\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
    from . import multiarray
ImportError: DLL load failed: 지정된 모듈을 찾을 수 없습니다.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\alkorang\.julia\packages\Conda\m7vem\deps\usr\lib\site-packages\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
  File "C:\Users\alkorang\.julia\packages\Conda\m7vem\deps\usr\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Users\alkorang\.julia\packages\Conda\m7vem\deps\usr\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Users\alkorang\.julia\packages\Conda\m7vem\deps\usr\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Users\alkorang\.julia\packages\Conda\m7vem\deps\usr\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: DLL load failed: 지정된 모듈을 찾을 수 없습니다.

>>>

[alkorang]

I remeber this issue didn't happen with Julia 0.6 and previous miniconda with Python 3.6.5. I cannot remeber the version of miniconda, though.

[alkorang]

I found the reason! DLLs are not included because Conda.BINDIR is not in PATH.

julia> using Conda, PyCall

julia> ENV["PATH"] = "$(Conda.BINDIR);$(ENV["PATH"])";

julia> py"""
       import numpy
       print(numpy.random.rand(1, 1))
       """
[[0.20880523]]

julia>

[simonbyrne]

Note that it's not just BINDIR that needs to be added to the PATH. I'm trying to set up RCall.jl to work with Conda, and it seems that other directories are also required:

julia> using Libdl, Conda

julia> libR = joinpath(Conda.ROOTENV, "Lib","R","Bin","x64","R.dll")
"C:\\Users\\IEUser\\.julia\\conda\\3\\Lib\\R\\Bin\\x64\\R.dll"

julia> dlopen(libR)
ERROR: could not load library "C:\Users\IEUser\.julia\conda\3\Lib\R\Bin\x64\R.dll"
The specified module could not be found.

Stacktrace:
 [1] #dlopen#3(::Bool, ::Function, ::String, ::UInt32) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Libdl\src\Libdl.jl:109
 [2] dlopen at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.1\Libdl\src\Libdl.jl:109 [inlined] (repeats 2 times)
 [3] top-level scope at none:0

julia> ENV["PATH"] = "$(joinpath(Conda.ROOTENV,"Library","mingw-w64","bin"));$(ENV["PATH"])"
"C:\\Users\\IEUser\\.julia\\conda\\3\\Library\\mingw-w64\\bin;C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2019.4.245\\windows\\mpi\\intel64\\bin;C:\\Program Files\\Microsoft MPI\\Bin\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\Puppet Labs\\Puppet\\bin;C:\\Users\\IEUser\\AppData\\Local\\Microsoft\\WindowsApps;"

julia> dlopen(libR)
Ptr{Nothing} @0x0000000029c00000

If I run the activate.bat script before starting Julia, I noticed the following directories are added to the PATH:

 "C:\\Users\\IEUser\\.julia\\conda\\3"
 "C:\\Users\\IEUser\\.julia\\conda\\3\\Library\\mingw-w64\\bin"
 "C:\\Users\\IEUser\\.julia\\conda\\3\\Library\\usr\\bin"
 "C:\\Users\\IEUser\\.julia\\conda\\3\\Library\\bin"
 "C:\\Users\\IEUser\\.julia\\conda\\3\\Scripts"
 "C:\\Users\\IEUser\\.julia\\conda\\3\\bin"
 "C:\\Users\\IEUser\\.julia\\conda\\3\\condabin"

It would be handy if there was a Conda.activate() function which would activate the appropriate paths.

[tkf]

It would be handy if there was a Conda.activate() function which would activate the appropriate paths.

I wrote it in #121 but I don't like that it'd be hard-coded in Conda.jl... It will be much cleaner if something like https://github.com/conda/conda/issues/6820#issuecomment-430847550 is implemented in conda. Meanwhile, maybe we can use something equivalent to conda run -n base env --null in Windows and copy $PATH from it.

[tkf]

Actually, I guess we can just do conda run -n base python -c 'import os; print(os.environ["PATH"], end="")' as python will be there in the path anyway.

[atbug]

Any update on this? I don't want to be stuck at python 3.6 for this.

[stevengj]

PyCall added the Conda bindir to its PATH in https://github.com/JuliaPy/PyCall.jl/pull/748

@simonbyrne, I would suggest that RCall do something similar for any paths that it needs.

I don't think that this is something that can really be done in the Conda.jl package per se because just because you do using Conda does not mean that you are actually using Conda libraries, just that you might be. On the other hand, I guess we could provide some kind of "activate" function that dependent modules could optionally call.

(I really hate that Windows requires you to modify a global environment variable for this, since potentially that has side effects on other code, but what can we do.)

[stevengj]

The other frustrating thing here is that the PATH mixes paths for DLL loading (which we want) with paths for executables (which we don't want). I'm worried about side effects on other codes if we indiscriminately extend the PATH.

[isuruf]

You could use the SetDllDirectoryA function if you want to avoid the executables

[stevengj]

I looked at SetDllDirectoryW and AddDllDirectory, but as described here it has a different precedence than adding a directory to the PATH. Also SetDllDirectoryW has the side effect of disabling "safe DLL" search mode, whereas the search order of AddDllDirectory paths is undefined.