JuliaPy / PyCall.jl

Package to call Python functions from the Julia language
MIT License
1.47k stars 190 forks source link

Can't build PyCall #704

Open SimonDanisch opened 5 years ago

SimonDanisch commented 5 years ago
 Installed PyCall ─ v1.91.2
  Updating `~/.julia/environments/v1.1/Project.toml`
  [438e738f] + PyCall v1.91.2
  Updating `~/.julia/environments/v1.1/Manifest.toml`
  [438e738f] + PyCall v1.91.2
  Building PyCall → `~/.julia/packages/PyCall/ttONZ/deps/build.log`
┌ Error: Error building `PyCall`: 
│   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
│                                  Dload  Upload   Total   Spent    Left  Speed
100 71.7M  100 71.7M    0     0  75.2M      0 --:--:-- --:--:-- --:--:-- 75.1M
│ PREFIX=/root/.julia/conda/3
│ Unpacking payload ...
Collecting package metadata (current_repodata.json): ...working... done                                   
│ Solving environment: ...working... done
│ 
│ ## Package Plan ##
│ 
│   environment location: /root/.julia/conda/3
│ 
│   added / updated specs:
│     - _libgcc_mutex==0.1=main
│     - asn1crypto==0.24.0=py37_0
│     - bzip2==1.0.8=h7b6447c_0
│     - ca-certificates==2019.5.15=0
│     - certifi==2019.6.16=py37_0
│     - cffi==1.12.3=py37h2e261b9_0
│     - chardet==3.0.4=py37_1
│     - conda-package-handling==1.3.11=py37_0
│     - conda==4.7.10=py37_0
│     - cryptography==2.7=py37h1ba5d50_0
│     - idna==2.8=py37_0
│     - libarchive==3.3.3=h5d8350f_5
│     - libedit==3.1.20181209=hc058e9b_0
│     - libffi==3.2.1=hd88cf55_4
│     - libgcc-ng==9.1.0=hdf63c60_0
│     - libstdcxx-ng==9.1.0=hdf63c60_0
│     - libxml2==2.9.9=hea5a465_1
│     - lz4-c==1.8.1.2=h14c3975_0
│     - lzo==2.10=h49e0be7_2
│     - ncurses==6.1=he6710b0_1
│     - openssl==1.1.1c=h7b6447c_1
│     - pip==19.1.1=py37_0
│     - pycosat==0.6.3=py37h14c3975_0
│     - pycparser==2.19=py37_0
│     - pyopenssl==19.0.0=py37_0
│     - pysocks==1.7.0=py37_0
│     - python-libarchive-c==2.8=py37_11
│     - python==3.7.3=h0371630_0
│     - readline==7.0=h7b6447c_5
│     - requests==2.22.0=py37_0
│     - ruamel_yaml==0.15.46=py37h14c3975_0
│     - setuptools==41.0.1=py37_0
│     - six==1.12.0=py37_0
│     - sqlite==3.29.0=h7b6447c_0
│     - tk==8.6.8=hbc83047_0
│     - tqdm==4.32.1=py_0
│     - urllib3==1.24.2=py37_0
│     - wheel==0.33.4=py37_0
│     - xz==5.2.4=h14c3975_4
│     - yaml==0.1.7=had09818_2
│     - zlib==1.2.11=h7b6447c_3
│     - zstd==1.3.7=h0b5b093_0
│ 
│ 
│ The following NEW packages will be INSTALLED:
│ 
│   _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
│   asn1crypto         pkgs/main/linux-64::asn1crypto-0.24.0-py37_0
│   bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
│   ca-certificates    pkgs/main/linux-64::ca-certificates-2019.5.15-0
│   certifi            pkgs/main/linux-64::certifi-2019.6.16-py37_0
│   cffi               pkgs/main/linux-64::cffi-1.12.3-py37h2e261b9_0
│   chardet            pkgs/main/linux-64::chardet-3.0.4-py37_1
│   conda              pkgs/main/linux-64::conda-4.7.10-py37_0
│   conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.3.11-py37_0
│   cryptography       pkgs/main/linux-64::cryptography-2.7-py37h1ba5d50_0
│   idna               pkgs/main/linux-64::idna-2.8-py37_0
│   libarchive         pkgs/main/linux-64::libarchive-3.3.3-h5d8350f_5
│   libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
│   libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
│   libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
│   libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
│   libxml2            pkgs/main/linux-64::libxml2-2.9.9-hea5a465_1
│   lz4-c              pkgs/main/linux-64::lz4-c-1.8.1.2-h14c3975_0
│   lzo                pkgs/main/linux-64::lzo-2.10-h49e0be7_2
│   ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
│   openssl            pkgs/main/linux-64::openssl-1.1.1c-h7b6447c_1
│   pip                pkgs/main/linux-64::pip-19.1.1-py37_0
│   pycosat            pkgs/main/linux-64::pycosat-0.6.3-py37h14c3975_0
│   pycparser          pkgs/main/linux-64::pycparser-2.19-py37_0
│   pyopenssl          pkgs/main/linux-64::pyopenssl-19.0.0-py37_0
│   pysocks            pkgs/main/linux-64::pysocks-1.7.0-py37_0
│   python             pkgs/main/linux-64::python-3.7.3-h0371630_0
│   python-libarchive~ pkgs/main/linux-64::python-libarchive-c-2.8-py37_11
│   readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
│   requests           pkgs/main/linux-64::requests-2.22.0-py37_0
│   ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.46-py37h14c3975_0
│   setuptools         pkgs/main/linux-64::setuptools-41.0.1-py37_0
│   six                pkgs/main/linux-64::six-1.12.0-py37_0
│   sqlite             pkgs/main/linux-64::sqlite-3.29.0-h7b6447c_0
│   tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
│   tqdm               pkgs/main/noarch::tqdm-4.32.1-py_0
│   urllib3            pkgs/main/linux-64::urllib3-1.24.2-py37_0
│   wheel              pkgs/main/linux-64::wheel-0.33.4-py37_0
│   xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
│   yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2
│   zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
│   zstd               pkgs/main/linux-64::zstd-1.3.7-h0b5b093_0
│ 
│ 
│ Preparing transaction: ...working... done
│ Executing transaction: ...working... WARNING conda.core.envs_manager:register_env(46): Unable to register environment. Path not writable or missing.
│   environment location: /root/.julia/conda/3
│   registry file: /root/.conda/environments.txt
│ done
│ installation finished.
│ Warning: 'defaults' already in 'channels' list, moving to the top
│ Collecting package metadata (current_repodata.json): ...working... done
│ Solving environment: ...working... done
│ 
│ ## Package Plan ##
│ 
│   environment location: /root/.julia/conda/3
│ 
│   added / updated specs:
│     - conda
│ 
│ 
│ The following packages will be downloaded:
│ 
│     package                    |            build
│     ---------------------------|-----------------
│     certifi-2019.6.16          |           py37_1         156 KB
│     chardet-3.0.4              |        py37_1003         173 KB
│     ------------------------------------------------------------
│                                            Total:         329 KB
│ 
│ The following packages will be UPDATED:
│ 
│   certifi                                  2019.6.16-py37_0 --> 2019.6.16-py37_1
│   chardet                                      3.0.4-py37_1 --> 3.0.4-py37_1003
│ 
│ 
│ 
│ Downloading and Extracting Packages
chardet-3.0.4        | 173 KB    | ########## | 100% 
certifi-2019.6.16    | 156 KB    | ########## | 100% 
│ Preparing transaction: ...working... done
│ Verifying transaction: ...working... done
│ Executing transaction: ...working... done
│ Collecting package metadata (current_repodata.json): ...working... done
│ Solving environment: ...working... done
│ 
│ ## Package Plan ##
│ 
│   environment location: /root/.julia/conda/3
│ 
│   added / updated specs:
│     - numpy
│ 
│ 
│ The following packages will be downloaded:
│ 
│     package                    |            build
│     ---------------------------|-----------------
│     blas-1.0                   |              mkl           6 KB
│     intel-openmp-2019.4        |              243         729 KB
│     libgfortran-ng-7.3.0       |       hdf63c60_0        1006 KB
│     mkl-2019.4                 |              243       131.2 MB
│     mkl_fft-1.0.12             |   py37ha843d7b_0         154 KB
│     mkl_random-1.0.2           |   py37hd81dba3_0         361 KB
│     numpy-1.16.4               |   py37h7e9f1db_0          48 KB
│     numpy-base-1.16.4          |   py37hde5b4d6_0         3.5 MB
│     ------------------------------------------------------------
│                                            Total:       137.0 MB
│ 
│ The following NEW packages will be INSTALLED:
│ 
│   blas               pkgs/main/linux-64::blas-1.0-mkl
│   intel-openmp       pkgs/main/linux-64::intel-openmp-2019.4-243
│   libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-7.3.0-hdf63c60_0
│   mkl                pkgs/main/linux-64::mkl-2019.4-243
│   mkl_fft            pkgs/main/linux-64::mkl_fft-1.0.12-py37ha843d7b_0
│   mkl_random         pkgs/main/linux-64::mkl_random-1.0.2-py37hd81dba3_0
│   numpy              pkgs/main/linux-64::numpy-1.16.4-py37h7e9f1db_0
│   numpy-base         pkgs/main/linux-64::numpy-base-1.16.4-py37hde5b4d6_0
│ 
│ 
│ 
│ Downloading and Extracting Packages
mkl-2019.4           | 131.2 MB  | ########## | 100% 
numpy-1.16.4         | 48 KB     | ########## | 100% 
blas-1.0             | 6 KB      | ########## | 100% 
libgfortran-ng-7.3.0 | 1006 KB   | ########## | 100% 
intel-openmp-2019.4  | 729 KB    | ########## | 100% 
numpy-base-1.16.4    | 3.5 MB    | ########## | 100% 
mkl_random-1.0.2     | 361 KB    | ########## | 100% 
mkl_fft-1.0.12       | 154 KB    | ########## | 100% 
│ Preparing transaction: ...working... done
│ Verifying transaction: ...working... done
│ Executing transaction: ...working... done
│ ┌ Info: Using the Python distribution in the Conda package by default.
│ └ To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
│ [ Info: Downloading miniconda installer ...
│ [ Info: Installing miniconda ...
│ [ Info: Running `conda config --add channels defaults --file /root/.julia/conda/3/condarc-julia.yml --force` in root environment
│ [ Info: Running `conda update -y conda` in root environment
│ [ Info: Running `conda install -y numpy` in root environment
│ ERROR: LoadError: IOError: could not spawn setenv(`/root/.julia/conda/3/bin/python /root/.julia/packages/PyCall/ttONZ/deps/find_libpython.py --list-all`,["WEBIO_WEBSOCKT_URL=wss://nextjournal.com/runner/LVTHeJHXjeqxJKHfBMVT6/runtime/c51cfdf9-861f-41a4-8779-7c673a8840d6/webio_websocket/", "PYTHONIOENCODING=UTF-8", "PATH=/usr/local/julia/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "WEBIO_SERVER_HOST_URL=0.0.0.0", "DISPLAY=:0", "JULIA_PATH=/usr/local/julia", "NVIDIA_DRIVER_CAPABILITIES=all", "LC_ALL=en_US.UTF-8", "JULIA_VERSION=1.1.0", "LANGUAGE=en_US.en", "NVIDIA_VISIBLE_DEVICES=void", "LANG=en_US.UTF-8", "JULIA_WEBIO_BASEURL=https://nextjournal.com/runner/LVTHeJHXjeqxJKHfBMVT6/runtime/c51cfdf9-861f-41a4-8779-7c673a8840d6", "BASH_ENV=/.bash_profile", "JULIA_LOAD_PATH=@:/tmp/tmpPtynLW", "NEXTJOURNAL_MOUNT_CUDA=9.2-cudnn7-devel-ubuntu18.04", "HOME=/root", "WEBIO_HTTP_PORT=9998", "OPENBLAS_MAIN_FREE=1", "HOSTNAME=f7c836db60d4", "GKSwstype=svg", "NEXTJOURNAL_RUNTIME_SERVICE_URL=https://nextjournal.com/runner/LVTHeJHXjeqxJKHfBMVT6/runtime/c51cfdf9-861f-41a4-8779-7c673a8840d6"]): invalid argument (EINVAL)
│ Stacktrace:
│  [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at ./process.jl:400
│  [2] setup_stdios(::getfield(Base, Symbol("##505#506")){Cmd}, ::Array{Any,1}) at ./process.jl:413
│  [3] _spawn at ./process.jl:412 [inlined]
│  [4] #eachline#511(::Bool, ::Function, ::Cmd) at ./process.jl:618
│  [5] eachline at ./process.jl:617 [inlined]
│  [6] #readlines#288 at ./io.jl:414 [inlined]
│  [7] readlines at ./io.jl:414 [inlined]
│  [8] exec_find_libpython at /root/.julia/packages/PyCall/ttONZ/deps/buildutils.jl:28 [inlined]
│  [9] #find_libpython#3(::typeof(Libdl.dlopen), ::Function, ::String) at /root/.julia/packages/PyCall/ttONZ/deps/buildutils.jl:43
│  [10] find_libpython(::String) at /root/.julia/packages/PyCall/ttONZ/deps/buildutils.jl:41
│  [11] top-level scope at /root/.julia/packages/PyCall/ttONZ/deps/build.jl:87
│  [12] include at ./boot.jl:326 [inlined]
│  [13] include_relative(::Module, ::String) at ./loading.jl:1038
│  [14] include(::Module, ::String) at ./sysimg.jl:29
│  [15] include(::String) at ./client.jl:403
│  [16] top-level scope at none:0
│ in expression starting at /root/.julia/packages/PyCall/ttONZ/deps/build.jl:43
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

I prepared a notebook, with some debug information and a way to reproduce it: https://nextjournal.com/sdanisch-scratch/pycall-build-error/ You can simply remix it, to have a look at whats happening. I have no idea what's going on - seems like everything is there and working properly, but somehow conda just doesn't want to parse its arguments... Also, I tried using the systems python3, which got me in different, but equally frustrating problems.

tkf commented 5 years ago

Strangely, it seems even /root/.julia/conda/3/bin/python --version fails with /bin/sh: 1: /root/.julia/conda/3/bin/python: Invalid argument:

https://nextjournal.com/a/LVdYo5T9yCxL5mtUAMShk?token=TrVSKpbecqZjYQSHjLjML2

But it looks like

;rm -rf /root/.julia/conda
]build PyCall

fixes the issue.