search

IntelPython / sdc

Numba extension for compiling Pandas data frames, Intel® Scalable Dataframe Compiler
https://intelpython.github.io/sdc-doc/
BSD 2-Clause "Simplified" License
645 stars 61 forks source link

Cannot solve condas environment #968

Closed LucyTishkina closed 3 years ago

LucyTishkina commented 3 years ago

I am trying to follow the instructions for installing binary packages using condas, but the process fails on the last step. When trying to execute conda install sdc -c intel/label/beta -c intel -c defaults -c conda-forge --override-channels I get a long list of conflicting packages. I am providing a partial output:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                     |                                                                                                                                                               -  

UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (pandas):

  - sdc -> pandas[version='0.25.3|1.0.5']
  - sdc -> pyarrow==0.17.0 -> pandas
  - sdc -> pyarrow[version='0.15.1|0.17.0']

The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libstdcxx-ng conflicts for:
six -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
sqlite -> libstdcxx-ng[version='>=4.9|>=7.3.0']
libstdcxx-ng
aws-c-common -> libstdcxx-ng[version='>=7.3.0|>=7.5.0']
numpy-base -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
sdc -> libstdcxx-ng[version='>=7.3.0']
pyarrow=2.0.0 -> libstdcxx-ng[version='>=7.3.0|>=7.5.0|>=9.3.0']
mkl_fft -> libstdcxx-ng[version='>=7.2.0|>=7.3.0']
python_abi -> python=3.9 -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
pandas=1.2.0 -> libstdcxx-ng[version='>=7.3.0|>=9.3.0']
glog -> gflags[version='>=2.2.2,<2.3.0a0'] -> libstdcxx-ng[version='>=7.5.0']
aws-c-event-stream -> libstdcxx-ng[version='>=7.3.0|>=9.3.0|>=7.5.0']
_openmp_mutex -> llvm-openmp[version='>=9.0.1'] -> libstdcxx-ng[version='>=7.3.0']
grpc-cpp -> libstdcxx-ng[version='>=7.3.0|>=9.3.0|>=7.5.0']
python[version='=3.7|3.6'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
numpy -> mkl_random[version='>=1.2.0,<2.0a0'] -> libstdcxx-ng[version='>=7.2.0|>=7.5.0|>=9.3.0']
pyarrow=2.0.0 -> boost-cpp -> libstdcxx-ng[version='>=4.9|>=7.2.0']
aws-checksums -> libstdcxx-ng[version='>=7.3.0|>=7.5.0']
wheel -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
abseil-cpp -> libstdcxx-ng[version='>=5.4.0|>=7.3.0|>=9.3.0|>=7.5.0']
re2 -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=7.5.0']
gflags -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0']
orc -> libstdcxx-ng[version='>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
pip -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
brotli -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
orc -> lz4-c[version='>=1.8.1.2,<1.9.0a0'] -> libstdcxx-ng[version='>=4.9']
libthrift -> libstdcxx-ng[version='>=7.3.0|>=9.3.0|>=7.5.0']
pandas=1.2.0 -> python[version='>=3.9,<3.10.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.5.0']
mkl_random -> libstdcxx-ng[version='>=7.2.0|>=7.3.0|>=7.5.0']
aws-sdk-cpp -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=9.3.0|>=7.5.0']
krb5 -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0']
zstd -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=9.3.0|>=7.5.0']
pytz -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
python-dateutil -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
mkl-service -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
certifi -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
numpy -> libstdcxx-ng[version='>=4.9|>=7.3.0']
snappy -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0']
setuptools -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
ncurses -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0']
libcurl -> krb5[version='>=1.17.1,<1.18.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0|>=9.3.0']
lz4-c -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=9.3.0|>=7.5.0']
mkl -> tbb=2021 -> libstdcxx-ng[version='>=4.9|>=7.3.0|>=9.3.0|>=7.5.0']
sdc -> numba==0.51.2 -> libstdcxx-ng[version='>=4.9|>=7.5.0|>=7.2.0|>=9.3.0']
mkl_fft -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.5.0|>=9.3.0']
sqlite -> ncurses[version='>=6.2,<7.0a0'] -> libstdcxx-ng[version='>=7.2.0|>=7.5.0']
libffi -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0']
glog -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0']
grpc-cpp -> gflags -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=5.4.0']
libprotobuf -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=9.3.0|>=7.5.0']
parquet-cpp -> arrow-cpp[version='>=0.13.0'] -> libstdcxx-ng[version='>=7.3.0|>=7.5.0|>=9.3.0']
mkl_random -> python[version='>=3.7,<3.8.0a0'] -> libstdcxx-ng[version='>=4.9|>=9.3.0']
blas -> llvm-openmp[version='>=9.0.1'] -> libstdcxx-ng[version='>=7.3.0']
libedit -> ncurses[version='>=6.2,<7.0a0'] -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0']
parquet-cpp -> libstdcxx-ng[version='>=4.9|>=7.2.0']
arrow-cpp -> libstdcxx-ng[version='>=4.9|>=7.2.0|>=7.3.0|>=7.5.0|>=9.3.0']
tomashek commented 3 years ago

Hello,

Using "conda install ..." attempts to update your existing environment, so conflicts may arise from your existing environment state. Please provide the output of "conda list" and "conda info", which will help us see what set of packages you are trying to add SDC to.

Alternatively, you can use "conda create -n sdc -c intel/label/beta -c intel -c defaults -c conda-forge --override-channels" to create a new environment. From there, you can see if there are obvious version conflicts with anything already in the environment that is failing.

kozlov-alexey commented 3 years ago

@Lusura Hi, thank you for reporting it!

This is caused by an early update of installation instruction. The sdc=0.38.0 binary that works with pyarrow=2.0.0 is not yet uploaded to intel/label/beta channel and current versions available there work only with pyarrow=0.17.0. It will probably be uploaded in the beginning of the next week and in the meantime you could either build SDC from sources following these steps: https://github.com/IntelPython/sdc#building-intel-sdc-from-source-on-linux https://github.com/IntelPython/sdc#building-intel-sdc-from-source-on-windows

or use sdc=0.37.0 and create environment with pyarrow=0.17.0 installed: > conda create -n sdc_env python=<3.7 or 3.6> pyarrow=0.17.0 pandas=1.2.0 -c anaconda -c conda-forge

LucyTishkina commented 3 years ago

I can install sdc 0.37.0 by downgrading the libraries, but when I import numba I get: double free or corruption (top) Aborted (core dumped)

kozlov-alexey commented 3 years ago

Can you try re-create the environment and see if it helps? That is do you have this problem when you execute:

conda create -n sdc-env python=<3.7 or 3.6> pyarrow=0.17.0 pandas=1.2.0 -c anaconda -c conda-forge conda activate sdc-env conda install sdc -c intel/label/beta -c intel -c defaults -c conda-forge --override-channels

If it's reproduced can you paste the output of conda list --revisions here?

LucyTishkina commented 3 years ago

I had to change pandas version to 1.0.5, I could not get the environment to resolve with 1.2.0 This is the output from conda list --revisions

2021-04-08 11:34:23  (rev 0)
    +abseil-cpp-20200225.2 (conda-forge/linux-64)
    +arrow-cpp-0.17.0 (conda-forge/linux-64)
    +aws-sdk-cpp-1.7.164 (conda-forge/linux-64)
    +blas-1.0 (anaconda/linux-64)
    +boost-cpp-1.72.0 (conda-forge/linux-64)
    +brotli-1.0.9 (anaconda/linux-64)
    +bzip2-1.0.8 (anaconda/linux-64)
    +c-ares-1.16.1 (anaconda/linux-64)
    +ca-certificates-2020.10.14 (anaconda/linux-64)
    +certifi-2020.6.20 (anaconda/linux-64)
    +curl-7.71.1 (anaconda/linux-64)
    +gflags-2.2.2 (anaconda/linux-64)
    +glog-0.4.0 (anaconda/linux-64)
    +grpc-cpp-1.28.1 (conda-forge/linux-64)
    +icu-68.1 (conda-forge/linux-64)
    +intel-openmp-2020.2 (anaconda/linux-64)
    +krb5-1.18.2 (anaconda/linux-64)
    +ld_impl_linux-64-2.33.1 (anaconda/linux-64)
    +libcurl-7.71.1 (anaconda/linux-64)
    +libedit-3.1.20191231 (anaconda/linux-64)
    +libevent-2.1.10 (conda-forge/linux-64)
    +libffi-3.3 (anaconda/linux-64)
    +libgcc-ng-9.1.0 (anaconda/linux-64)
    +libprotobuf-3.11.4 (anaconda/linux-64)
    +libssh2-1.9.0 (anaconda/linux-64)
    +libstdcxx-ng-9.1.0 (anaconda/linux-64)
    +libthrift-0.13.0 (conda-forge/linux-64)
    +lz4-c-1.9.2 (anaconda/linux-64)
    +mkl-2019.4 (anaconda/linux-64)
    +mkl-service-2.3.0 (anaconda/linux-64)
    +mkl_fft-1.2.0 (anaconda/linux-64)
    +mkl_random-1.1.0 (anaconda/linux-64)
    +ncurses-6.2 (anaconda/linux-64)
    +numpy-1.19.1 (anaconda/linux-64)
    +numpy-base-1.19.1 (anaconda/linux-64)
    +openssl-1.1.1h (anaconda/linux-64)
    +pandas-1.0.5 (anaconda/linux-64)
    +parquet-cpp-1.5.1 (conda-forge/noarch)
    +pip-20.2.4 (anaconda/linux-64)
    +pyarrow-0.17.0 (conda-forge/linux-64)
    +python-3.7.9 (anaconda/linux-64)
    +python-dateutil-2.8.1 (anaconda/noarch)
    +python_abi-3.7 (conda-forge/linux-64)
    +pytz-2020.1 (anaconda/noarch)
    +re2-2019.08.01 (anaconda/linux-64)
    +readline-8.0 (anaconda/linux-64)
    +setuptools-50.3.0 (anaconda/linux-64)
    +six-1.15.0 (anaconda/noarch)
    +snappy-1.1.8 (anaconda/linux-64)
    +sqlite-3.33.0 (anaconda/linux-64)
    +thrift-compiler-0.13.0 (conda-forge/linux-64)
    +thrift-cpp-0.13.0 (conda-forge/linux-64)
    +tk-8.6.10 (anaconda/linux-64)
    +wheel-0.35.1 (anaconda/noarch)
    +xz-5.2.5 (anaconda/linux-64)
    +zlib-1.2.11 (anaconda/linux-64)
    +zstd-1.4.5 (anaconda/linux-64)

2021-04-08 11:34:43  (rev 1)
     ca-certificates  {2020.10.14 (anaconda/linux-64) -> 2020.12.5 (intel/linux-64)}
     certifi  {2020.6.20 (anaconda/linux-64) -> 2020.12.5 (intel/linux-64)}
     openssl  {1.1.1h (anaconda/linux-64) -> 1.1.1h (intel/linux-64)}
    +icc_rt-2020.3 (intel/linux-64)
    +intelpython-2021.1.1 (intel/linux-64)
    +libllvm10-10.0.1 (defaults/linux-64)
    +llvmlite-0.34.0 (defaults/linux-64)
    +numba-0.51.2 (intel/linux-64)
    +sdc-0.37.0 (intel/label/beta/linux-64)
    +tbb-2021.2.0 (intel/linux-64)
    +tbb4py-2021.2.0 (intel/linux-64)
kozlov-alexey commented 3 years ago

Yes, sorry, that's probably because sdc=0.37 has hard dependency on pandas=1.0.5. We will update the install instruction, since all the requirements are now kept up-to-date in conda-recipe/meta.yaml so there's actually no need to specify them on the command line. Also AFAIK sdc=0.38 was finally published to intel/label/beta, so installing it should work now.