search

conda-forge / pyarrow-feedstock

A conda-smithy repository for pyarrow.
BSD 3-Clause "New" or "Revised" License
6 stars 26 forks source link

ImportError: DLL load failed: The specified module could not be found #54

Closed dhirschfeld closed 6 years ago

dhirschfeld commented 6 years ago

The dreaded DLL load failed error :(

This is on win64 with py36 and the latest pyarrow. I tested that this also happens in a new environment:

λ conda create -n arw python=3.6 defaults::numpy conda-forge::arrow-cpp conda-forge::parquet-cpp conda-forge::pyarrow conda-forge::vc --yes
Solving environment: done

## Package Plan ##

  environment location: C:\Miniconda3\envs\arw

  added / updated specs:
    - conda-forge::arrow-cpp
    - conda-forge::parquet-cpp
    - conda-forge::pyarrow
    - conda-forge::vc
    - defaults::numpy
    - python=3.6

The following NEW packages will be INSTALLED:

    arrow-cpp:       0.10.0-py36_vc14h12fa3ca_0 conda-forge [vc14]
    blas:            1.0-mkl
    boost-cpp:       1.67.0-hfa6e2cd_4
    certifi:         2018.8.13-py36_0
    icc_rt:          2018.0.2-0                 numba
    intel-openmp:    2018.0.3-0
    libboost:        1.67.0-hfd51bdf_4
    mkl:             2018.0.3-1
    mkl_fft:         1.0.6-py36_0               conda-forge
    mkl_random:      1.0.1-py36h77b88f5_1
    numpy:           1.15.0-py36h9fa60d3_0
    numpy-base:      1.15.0-py36h4a99626_0
    pandas:          0.23.4-py36h830ac7b_0
    parquet-cpp:     1.5.0.pre-h037026a_0       conda-forge
    pip:             18.0-py36_1                conda-forge
    pyarrow:         0.10.0-py36h6538335_0      conda-forge
    python:          3.6.6-hea74fb7_0
    python-dateutil: 2.7.3-py36_0
    pytz:            2018.5-py36_0
    setuptools:      40.0.0-py36_0
    six:             1.11.0-py36_1
    snappy:          1.1.7-h777316e_3
    vc:              14-0                       conda-forge
    vs2015_runtime:  15.5.2-3
    wheel:           0.31.1-py36_0
    wincertstore:    0.2-py36h7fe50ca_0
    xz:              5.2.4-h2fa13f4_4
    zlib:            1.2.11-h8395fce_2

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Results in a broken environment:

λ activate arw
(arw) λ  python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Miniconda3\envs\arw\lib\site-packages\pyarrow\__init__.py", line 60, in <module>
    from pyarrow.lib import cpu_count, set_cpu_count
ImportError: DLL load failed: The specified module could not be found.
>>>
xhochy commented 6 years ago

Seems like you get libboost and boost-cpp from defaults. Can you try this here with them from conda-forge?

The best would be to use as few as possible packages from defaults with conda-forge packages. With the compiler migration in place, this is a source of many errors.

dhirschfeld commented 6 years ago

The best would be to use as few as possible packages from defaults with conda-forge packages. With the compiler migration in place, this is a source of many errors.

Yeah, this is a cause of much pain at present. My default setup is to have defaults, conda-forge and dhirschfeld channels with channel_priority: false so that I pick the latest package irrespective of what channel it is in. This has been causing a number of such problems recently though.

In an ideal world conda would be able to distinguish abi incompatible packages and choose between them but we're not there yet.

dhirschfeld commented 6 years ago

Hmm, the update plan seems to want to downgrade me:

(arw) λ conda install conda-forge::boost-cpp --yes
<snip>
The following packages will be UPDATED:

    boost-cpp:   1.67.0-hfa6e2cd_4                      --> 1.68.0-h6a4c333_0 conda-forge
    certifi:     2018.8.13-py36_0           conda-forge --> 2018.8.13-py36_0

The following packages will be DOWNGRADED:

    arrow-cpp:   0.10.0-py36_vc14h12fa3ca_0 conda-forge [vc14] --> 0.8.0-py36_vc14_4 conda-forge [vc14]
    parquet-cpp: 1.5.0.pre-h037026a_0       conda-forge --> 1.4.0.pre-vc14_2  conda-forge [vc14]
    pyarrow:     0.10.0-py36h6538335_0      conda-forge --> 0.8.0-py36_vc14_0 conda-forge [vc14]

Currently investigating...

Edit: Updating pyarrow did the trick:

(arw) λ conda update pyarrow

The following packages will be UPDATED:

    arrow-cpp:   0.8.0-py36_vc14_4 conda-forge [vc14] --> 0.10.0-py36_vc14h12fa3ca_0 conda-forge [vc14]
    parquet-cpp: 1.4.0.pre-vc14_2  conda-forge [vc14] --> 1.5.0.pre-h037026a_0       conda-forge
    pyarrow:     0.8.0-py36_vc14_0 conda-forge [vc14] --> 0.10.0-py36h6538335_0      conda-forge

The following packages will be DOWNGRADED:

    boost-cpp:   1.68.0-h6a4c333_0 conda-forge --> 1.67.0-hea38baa_0          conda-forge

...so it seems the newer pyarrow depends on the older boost-cpp.

arrow-cpp 0.8.0 py36_vc14_4
---------------------------
name        : arrow-cpp
version     : 0.8.0
build string: py36_vc14_4
build number: 4
size        : 3.7 MB
arch        : x86_64
constrains  : ()
features    : vc14
license     : Apache 2.0
platform    : win32
subdir      : win-64
dependencies:
    numpy >=1.12
    python 3.6*
    snappy
    vc 14.*

arrow-cpp 0.10.0 py36_vc14h12fa3ca_0
------------------------------------
name        : arrow-cpp
version     : 0.10.0
build string: py36_vc14h12fa3ca_0
build number: 0
size        : 35.8 MB
arch        : x86_64
constrains  : ()
features    : vc14
license     : Apache 2.0
platform    : win32
subdir      : win-64
timestamp   : 1533594888770
dependencies:
    boost-cpp >=1.67.0,<1.67.1.0a0
    numpy >=1.12,<2.0a0
    python >=3.6,<3.7.0a0
    snappy >=1.1.7,<1.1.8.0a0
    vc 14.*
dhirschfeld commented 6 years ago

Thanks @xhochy - that was it! libboost is only required by the defaults package so just installing boost-cpp from conda-forge was sufficient to avoid this problem.

I'll close this as resolved. If anyone else hits this problem they can still find this solution by searching the issues.

marcco411 commented 5 years ago

Hi, I am having the exact same error message as stated above:

Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 19:01:41) [MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information.

import pyarrow Traceback (most recent call last): File "", line 1, in File "D:\Python\Miniconda3\lib\site-packages\pyarrow__init__.py", line 54, in from pyarrow.lib import cpu_count, set_cpu_count ImportError: DLL load failed: The specified module could not be found.

but for some reason all the steps mentioned above do not work for me. any other idea of what could go wrong?

xhochy commented 5 years ago

@marcco411 Can you open a new issue on https://issues.apache.org/jira/projects/ARROW with all the steps to reproduce your problem incl. the log output of the conda commands? This might be a different issue.

akbaritabar commented 4 years ago

Today I had the same issue on Windows platform, which came after "conda update --all". I had "Pyarrow" installed from PIP. I couldn't resolve it by re-installing pyarrow, or removing then installing from conda-forge.

What solved the issue was to remove the whole Anaconda and reinstall again (maybe not necessary in your case), then I started using conda virtual environments for the first time (noob here, I know) and then installing everything from conda-forge in a clean environment solved the issue.