search

wmayner / pyemd

Fast EMD for Python: a wrapper for Pele and Werman's C++ implementation of the Earth Mover's Distance metric
MIT License
479 stars 62 forks source link

pyemd in databricks could not be installed #37

Closed lynochka closed 1 year ago

lynochka commented 5 years ago

pyemd in databricks could not be installed in Databrick Runtime version 4.2 (no workers). The error is below:

%sh
lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Stepping:              1
CPU MHz:               2294.685
BogoMIPS:              4589.37
Hypervisor vendor:     Microsoft
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              51200K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt

Python:

sys.version_info
Out[25]: sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)

Pip:

%sh 
/databricks/python3/bin/python3 -m pip freeze
ansi2html==1.1.1
azure-common==1.1.18
azure-nspkg==3.0.2
azure-storage-blob==1.1.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
backports.shutil-get-terminal-size==1.0.0
boto==2.42.0
boto3==1.4.1
botocore==1.4.70
brewer2mpl==1.4.1
certifi==2016.2.28
cffi==1.7.0
chardet==2.3.0
colorama==0.3.7
configobj==5.0.6
cryptography==1.5
cycler==0.10.0
Cython==0.24.1
decorator==4.0.10
docutils==0.14
enum34==1.1.6
et-xmlfile==1.0.1
freetype-py==1.0.2
funcsigs==1.0.2
fusepy==2.0.4
ggplot==0.6.8
html5lib==0.999
idna==2.1
ipaddress==1.0.16
ipython==2.2.0
ipython-genutils==0.1.0
jdcal==1.2
Jinja2==2.8
jmespath==0.9.0
llvmlite==0.13.0
lxml==3.6.4
MarkupSafe==0.23
matplotlib==1.5.3
mpld3==0.2
msgpack-python==0.4.7
ndg-httpsclient==0.3.3
numba==0.28.1
numpy==1.16.1
openpyxl==2.3.2
pandas==0.24.1
pathlib2==2.1.0
patsy==0.4.1
pexpect==4.0.1
pickleshare==0.7.4
Pillow==3.3.1
pkg-resources==0.0.0
ply==3.9
prompt-toolkit==1.0.7
psycopg2==2.6.2
ptyprocess==0.5.1
py4j==0.10.3
pyarrow==0.8.0
pyasn1==0.1.9
pycparser==2.14
pycurl==7.43.0
Pygments==2.1.3
pygobject==3.20.0
pyOpenSSL==16.0.0
pyparsing==2.2.0
pypng==0.0.18
python-apt==1.1.0b1+ubuntu0.16.4.1
python-dateutil==2.5.3
python-geohash==0.8.5
pytz==2016.6.1
requests==2.11.1
s3transfer==0.1.9
scikit-learn==0.18.1
scipy==0.18.1
scour==0.32
seaborn==0.7.1
simplejson==3.8.2
singledispatch==3.4.0.3
six==1.10.0
ssh-import-id==5.5
statsmodels==0.8.0
traitlets==4.3.0
unattended-upgrades==0.1
urllib3==1.19.1
virtualenv==15.0.1
wcwidth==0.1.7

Error:

%sh
/databricks/python3/bin/python3 -m pip install pyemd
  Downloading https://files.pythonhosted.org/packages/c0/c5/7fea8e7a71cd026b30ed3c40e4c5ea13a173e28f8855da17e25271e8f545/pyemd-0.5.1.tar.gz (91kB)
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/pyemd.egg-info
    writing requirements to pip-egg-info/pyemd.egg-info/requires.txt
    writing pip-egg-info/pyemd.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/pyemd.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/pyemd.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/pyemd.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-eii9t9ra/pyemd/setup.py", line 121, in <module>
        'Programming Language :: Python :: 3.6',
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 278, in run
        self.find_sources()
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 293, in find_sources
        mm.run()
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 524, in run
        self.add_defaults()
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 560, in add_defaults
        sdist.add_defaults(self)
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/usr/lib/python3.5/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/lib/python3.5/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-install-eii9t9ra/pyemd/setup.py", line 77, in finalize_options
        import numpy
    ImportError: No module named 'numpy'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-eii9t9ra/pyemd/

Might it be that the path to python in /usr/lib or similar is hardcoded somewhere there?

wmayner commented 5 years ago

The problem is that NumPy needs to be imported during the setup process so that its C headers can be included, and for some reason it cannot be imported.

You could try reinstalling NumPy directly before installing PyEMD.

matallanas commented 5 years ago

It is still not working, I tried installing first Numpy and then PyEMD but it doesn't work. Has anyone tried to install it successfully? Thank you very much.