search

JuliaPy / Conda.jl

Conda managing Julia binary dependencies
Other
173 stars 57 forks source link

Fails to find Python 3.7 interpreter on NixOS 18.09, despite it being in PATH #134

Open fusion809 opened 5 years ago

fusion809 commented 5 years ago

Hi,

I've been trying to install IJulia using Julia's package manager, and Conda is a dependency of it that doesn't seem to want to cooperate with me. This is the error I'm getting:

┌ Error: Error building `IJulia`: 
│   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
│                                  Dload  Upload   Total   Spent    Left  Speed
100 59.6M  100 59.6M    0     0  5207k      0  0:00:11  0:00:11 --:--:-- 5199k
│ PREFIX=/home/fusion809/.julia/conda/3
│ reinstalling: python-3.7.0-hc3d631a_0 ...
│ /home/fusion809/.julia/conda/3/installer.sh: line 370: /home/fusion809/.julia/conda/3/pkgs/python-3.7.0-hc3d631a_0/bin/python: No such file or directory
│ ┌ Warning: Could not execute `jupyter --version`.
│ └ @ Main ~/.julia/packages/IJulia/DL02A/deps/build.jl:41
│ [ Info: Installing Jupyter via the Conda package.
│ [ Info: Downloading miniconda installer ...
│ [ Info: Installing miniconda ...
│ ERROR: LoadError: failed process: Process(`/home/fusion809/.julia/conda/3/installer.sh -b -f -p /home/fusion809/.julia/conda/3`, ProcessExited(1)) [1]
│ Stacktrace:

/home/fusion809/.julia/conda/3/pkgs/python-3.7.0-hc3d631a_0/bin/python exists, it is symlink to python37, which is a command in PATH. Although, I should mention NixOS places binaries in non-FHS locations, and while my user PATH environment variable points to ~/.nix-profile/bin, where python3.7 is found, it is not in the usual FHS location of /usr/bin.

Thanks for your time, Brenton

tkf commented 5 years ago

Maybe it's helpful to see the output of

cd ~/.julia/conda/3/pkgs/python-3.7.0-hc3d631a_0/bin/
ls -lh python*

I happened to have the same package in the cache:

$ cd ~/miniconda3/pkgs/python-3.7.0-hc3d631a_0/bin
$ ls -lh python*
lrwxrwxrwx 1 takafumi takafumi    9 Jul 29 15:11 python -> python3.7
lrwxrwxrwx 1 takafumi takafumi    9 Jul 29 15:11 python3 -> python3.7
-rwxrwxr-x 1 takafumi takafumi 3.8M Jun 28 06:29 python3.7
lrwxrwxrwx 1 takafumi takafumi   17 Jul 29 15:11 python3.7-config -> python3.7m-config
lrwxrwxrwx 1 takafumi takafumi    9 Jul 29 15:11 python3.7m -> python3.7
-rwxrwxr-x 1 takafumi takafumi 3.5K Jun 28 06:29 python3.7m-config
lrwxrwxrwx 1 takafumi takafumi   17 Jul 29 15:11 python3-config -> python3.7m-config

You may also want to check if manually installing miniconda works. It's not clear if the issue is in Conda.jl.

fusion809 commented 5 years ago 
lrwxrwxrwx 1 fusion809 users    9 Jun 28 23:29 python -> python3.7
lrwxrwxrwx 1 fusion809 users    9 Jun 28 23:29 python3 -> python3.7
-rwxr-xr-x 1 fusion809 users 3.8M Jun 28 23:29 python3.7
lrwxrwxrwx 1 fusion809 users   17 Jun 28 23:29 python3.7-config -> python3.7m-config
lrwxrwxrwx 1 fusion809 users    9 Jun 28 23:29 python3.7m -> python3.7
-rwxr-xr-x 1 fusion809 users 3.5K Jun 28 23:29 python3.7m-config
lrwxrwxrwx 1 fusion809 users   17 Jun 28 23:29 python3-config -> python3.7m-config
fusion809 commented 5 years ago

How would I check if miniconda works? I'm not familiar with it.

tkf commented 5 years ago

So python3.7 exists. I wonder why the installer script says it doesn't.

If you want to investigate more, here is the installation guide of miniconda: https://conda.io/docs/user-guide/install/linux.html

fusion809 commented 5 years ago

(My deleted comment was when I forgot I was on Arch, instead of NixOS) Turns out I get the same error with it:

./Miniconda3-latest-Linux-x86_64.sh: line 370: /home/fusion809/miniconda3/pkgs/python-3.7.0-hc3d631a_0/bin/python: No such file or directory
fusion809 commented 5 years ago

I have reported this upstream at conda/conda#8036.

tkf commented 5 years ago

Thanks for reporting it to upstream!

I noticed nix has conda as a package https://github.com/NixOS/nixpkgs/issues/26245. Maybe you can install it this way and point environment variable CONDA_JL_HOME to it https://github.com/JuliaPy/Conda.jl#using-a-pre-existing-conda-installation

fusion809 commented 5 years ago

Thanks mate for trying to help me, although Python packages, especially Python package managers, on NixOS 18.09 have a bad habit of biting me in the rear end. Largely because using them the way you do on most Linux distros tends to return errors relating to the non-FHS compliance of NixOS. I cannot use i3pystatus on NixOS due to these issues, I've tried following their Wiki but pip and even their own i3pystatus package do not seem to want to play ball. The conda command itself throws this error on it: https://gist.github.com/fusion809/52f9d67f99aa96bc40780ca072a02550.

tkf commented 5 years ago

Hmm... That's too bad. What is bizarre is that the installation script says pkgs/python-3.7.0-hc3d631a_0/bin/python does not exists when it does. I also don't see why it would be related to non-FHS layout. Anyway, let's hope conda devs can help.

jlou2u commented 5 years ago

I think this is because miniconda expects libraries in /lib64 and possibly different versions. I get similar results:

b818679acc42:/# bash ~/miniconda.sh -b -p /opt/conda -s                                                                                                                                     
PREFIX=/opt/conda                                                                                                                                                                           
installing: python-3.7.0-hc3d631a_0 ...                                                                                                                                                     
/root/miniconda.sh: line 370: /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: No such file or directory             
b818679acc42:/# ldd /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python                                                                                                                      
        /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                                        
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                     
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                           
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                          
        libutil.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                        
        librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                          
        libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f6e26639000)                                                                                                                           
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __snprintf_chk: symbol not found                                                                                       
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __wcscat_chk: symbol not found                                                                                         
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __isnan: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __finite: symbol not found                                                                                             
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __isinf: symbol not found
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __isinff: symbol not found                                                                                             
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __realpath_chk: symbol not found                                                                                       
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __memmove_chk: symbol not found                                                                                        
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __memcpy_chk: symbol not found                                                                                         
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __wcsncpy_chk: symbol not found                                                                                        
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __wcscpy_chk: symbol not found                                                                                         
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __rawmemchr: symbol not found                                                                                          
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __open64_2: symbol not found                                                                                           
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __sched_cpufree: symbol not found                                                                                      
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __sched_cpualloc: symbol not found                                                                                     
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __vsnprintf_chk: symbol not found                                                                                      
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __strncpy_chk: symbol not found                                                                                        
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __strcpy_chk: symbol not found                                                                                         
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __printf_chk: symbol not found                                                                                         
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __fread_chk: symbol not found                                                                                          
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __fprintf_chk: symbol not found                                                                                        
Error relocating /opt/conda/pkgs/python-3.7.0-hc3d631a_0/bin/python: __sprintf_chk: symbol not found

I think this is solved by people using buildFHSUserEnv, ie: https://github.com/NixOS/nixpkgs/blob/db558b31b9af9db59a8000c85c21057c88efcc25/pkgs/tools/package-management/conda/default.nix#L55

I don't understand why buildFHSUserEnv is required instead of being able to use patchelf to just repoint things, but that's what I'm trying to understand now.

Hopefully this helps.

fusion809 commented 5 years ago

Yeah but buildFHSUserEnv is only usable when you're building a package, not when you're using NixOS and trying to install Conda.jl with Julia.

sbourdeauducq commented 5 years ago

I don't understand why buildFHSUserEnv is required instead of being able to use patchelf to just repoint things, but that's what I'm trying to understand now.

The conda installer extracts the Python ELF binary and expects to be able to run it immediately, if you want to insert patchelf, you need to modify the conda installer (and you'd also have to use patchelf on every conda package you install that contains ELFs - it gets difficult).

ahumenberger commented 4 years ago

Any progress here? I'm experiencing the same problem with Julia 1.0.4 on Ubuntu 18.04 LTS and 19.04.

julia> Conda.list()
[ Info: Downloading miniconda installer ...
--2019-10-02 08:53:06--  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86.sh
Resolving repo.continuum.io (repo.continuum.io)... 104.18.200.79, 104.18.201.79, 2606:4700::6812:c94f, ...
Connecting to repo.continuum.io (repo.continuum.io)|104.18.200.79|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65741329 (63M) [application/x-sh]
Saving to: ‘/home/ubuntu/.julia/conda/3/installer.sh’

/home/ubuntu/.julia 100%[===================>]  62,70M  9,87MB/s    in 6,4s    

2019-10-02 08:53:12 (9,81 MB/s) - ‘/home/ubuntu/.julia/conda/3/installer.sh’ saved [65741329/65741329]

[ Info: Installing miniconda ...
PREFIX=/home/ubuntu/.julia/conda/3
reinstalling: python-3.7.1-h0371630_7 ...
/home/ubuntu/.julia/conda/3/installer.sh: 373: /home/ubuntu/.julia/conda/3/installer.sh: /home/ubuntu/.julia/conda/3/pkgs/python-3.7.1-h0371630_7/bin/python: not found
ERROR: failed process: Process(`/home/ubuntu/.julia/conda/3/installer.sh -b -f -p /home/ubuntu/.julia/conda/3`, ProcessExited(1)) [1]
Stacktrace:
 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
 [2] pipeline_error at ./process.jl:705 [inlined]
 [3] #run#505(::Bool, ::Function, ::Cmd) at ./process.jl:663
 [4] run at ./process.jl:661 [inlined]
 [5] _install_conda(::String, ::Bool) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:165
 [6] _install_conda(::String) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:152
 [7] runconda(::Cmd, ::String) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:111
 [8] list(::String) at /home/ubuntu/.julia/packages/Conda/kLXeC/src/Conda.jl:225 (repeats 2 times)
 [9] top-level scope at none:0
ahumenberger commented 4 years ago

Any progress here? I'm experiencing the same problem with Julia 1.0.4 on Ubuntu 18.04 LTS and 19.04.

I solved my issue. It turned out that I had a 32-bit Julia installed on a 64-bit Ubuntu, and Conda.jl tried to install a 32-bit miniconda which failed. Using a 64-bit Julia works.

natema commented 4 years ago

Thanks @ahumenberger, I installed Julia via snap and was having this issue, but reinstallling the correct Julia binaries solved the problem.