vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.3k stars 590 forks source link

[BUG-REPORT] #1178

Closed komskisp closed 3 years ago

komskisp commented 3 years ago

Before typing this, I have tried multiple ways to install Vaex for last two days. did it through pip install, virtual env. Pip setup tools are upgraded pip is upgraded Python 3.9.

Tried cloning the git repo, errors. Tried pip install errors. Tried approach for developers errors. Tried installing one package at a time: Vaex-core, Vaex-arrow tried --upgrade Vaex.

These are the mistakes it spits out: Any help much appreciated.

(env) pauliuskomskis@Pauliuss-Air Desktop % pip install vaex-core Collecting vaex-core Using cached vaex-core-2.0.3.tar.gz (2.1 MB) Requirement already satisfied: numpy>=1.16 in ./env/lib/python3.9/site-packages (from vaex-core) (1.19.5) Requirement already satisfied: astropy>=2 in ./env/lib/python3.9/site-packages (from vaex-core) (4.2) Requirement already satisfied: aplus in ./env/lib/python3.9/site-packages (from vaex-core) (0.11.0) Requirement already satisfied: tabulate>=0.8.3 in ./env/lib/python3.9/site-packages (from vaex-core) (0.8.7) Requirement already satisfied: future>=0.15.2 in ./env/lib/python3.9/site-packages (from vaex-core) (0.18.2) Requirement already satisfied: pyyaml in ./env/lib/python3.9/site-packages (from vaex-core) (5.4) Requirement already satisfied: progressbar2 in ./env/lib/python3.9/site-packages (from vaex-core) (3.53.1) Requirement already satisfied: psutil>=1.2.1 in ./env/lib/python3.9/site-packages (from vaex-core) (5.8.0) Requirement already satisfied: requests in ./env/lib/python3.9/site-packages (from vaex-core) (2.25.1) Requirement already satisfied: six in ./env/lib/python3.9/site-packages (from vaex-core) (1.15.0) Requirement already satisfied: cloudpickle in ./env/lib/python3.9/site-packages (from vaex-core) (1.6.0) Requirement already satisfied: pandas in ./env/lib/python3.9/site-packages (from vaex-core) (1.2.0) Requirement already satisfied: dask[array] in ./env/lib/python3.9/site-packages (from vaex-core) (2021.1.0) Requirement already satisfied: nest-asyncio>=1.3.3 in ./env/lib/python3.9/site-packages (from vaex-core) (1.4.3) Requirement already satisfied: pyerfa in ./env/lib/python3.9/site-packages (from astropy>=2->vaex-core) (1.7.1.1) Requirement already satisfied: toolz>=0.8.2 in ./env/lib/python3.9/site-packages (from dask[array]->vaex-core) (0.11.1) Requirement already satisfied: python-dateutil>=2.7.3 in ./env/lib/python3.9/site-packages (from pandas->vaex-core) (2.8.1) Requirement already satisfied: pytz>=2017.3 in ./env/lib/python3.9/site-packages (from pandas->vaex-core) (2020.5) Requirement already satisfied: python-utils>=2.3.0 in ./env/lib/python3.9/site-packages (from progressbar2->vaex-core) (2.5.1) Requirement already satisfied: chardet<5,>=3.0.2 in ./env/lib/python3.9/site-packages (from requests->vaex-core) (4.0.0) Requirement already satisfied: idna<3,>=2.5 in ./env/lib/python3.9/site-packages (from requests->vaex-core) (2.10) Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.9/site-packages (from requests->vaex-core) (1.26.2) Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.9/site-packages (from requests->vaex-core) (2020.12.5) Using legacy 'setup.py install' for vaex-core, since package 'wheel' is not installed. Installing collected packages: vaex-core Running setup.py install for vaex-core ... error ERROR: Command errored out with exit status 1: command: /Users/pauliuskomskis/Desktop/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/setup.py'"'"'; file='"'"'/private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-record-_o4_m3uz/install-record.txt --single-version-externally-managed --compile --install-headers /Users/pauliuskomskis/Desktop/env/include/site/python3.9/vaex-core cwd: /private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/ Complete output (186 lines): /private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/setup.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp running install running build running build_py creating build creating build/lib.macosx-10.9-x86_64-3.9 creating build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/scopes.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/geo.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/serialize.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/functions.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/misc_cmdline.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/promise.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/expresso.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/execution.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/grids.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/tasks.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/version.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/encoding.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/benchmark.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/legacy.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/dataset_mmap.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/_version.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/samp.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/events.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/dataframe.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/asyncio.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/cpu.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/datasets.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/init.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/dataset_utils.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/formatting.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/selections.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/export.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/dataset.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/hash.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/delayed.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/utils.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/kld.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/stat.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/parallelize.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/settings.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/array_types.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/agg.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/groupby.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/expression.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/json.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/image.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/main.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/strings.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/column.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/multithreading.py -> build/lib.macosx-10.9-x86_64-3.9/vaex copying vaex/meta.py -> build/lib.macosx-10.9-x86_64-3.9/vaex creating build/lib.macosx-10.9-x86_64-3.9/vaex/core copying vaex/core/_version.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/core copying vaex/core/init.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/core creating build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/colfits.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/cache.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/init.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/other.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/gadget.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/s3.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file copying vaex/file/column.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/file creating build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/misc.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/expresso.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/plot.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/ui.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/init.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/all.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/dataset.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/main.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test copying vaex/test/cmodule.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/test creating build/lib.macosx-10.9-x86_64-3.9/vaex/ext copying vaex/ext/ipyvolume.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/ext copying vaex/ext/jprops.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/ext copying vaex/ext/readcol.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/ext copying vaex/ext/init.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/ext copying vaex/ext/common.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/ext copying vaex/ext/bokeh.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/ext creating build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/ordereddict.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/samp.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/expressions.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/init.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/pandawrap.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/progressbar.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc copying vaex/misc/parallelize.py -> build/lib.macosx-10.9-x86_64-3.9/vaex/misc creating build/lib.macosx-10.9-x86_64-3.9/vaex/test/files copying vaex/test/files/gaia-small-colfits-basic.fits -> build/lib.macosx-10.9-x86_64-3.9/vaex/test/files copying vaex/test/files/gaia-small-fits-basic.fits -> build/lib.macosx-10.9-x86_64-3.9/vaex/test/files copying vaex/test/files/gaia-small-colfits-plus.fits -> build/lib.macosx-10.9-x86_64-3.9/vaex/test/files copying vaex/test/files/gaia-small-fits-plus.fits -> build/lib.macosx-10.9-x86_64-3.9/vaex/test/files copying vaex/test/files/gaia-small-votable.vot -> build/lib.macosx-10.9-x86_64-3.9/vaex/test/files copying vaex/test/files/default_amuse_plummer.hdf5 -> build/lib.macosx-10.9-x86_64-3.9/vaex/test/files running build_ext building 'vaex.vaexfast' extension creating build/temp.macosx-10.9-x86_64-3.9 creating build/temp.macosx-10.9-x86_64-3.9/src gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Users/pauliuskomskis/Desktop/env/lib/python3.9/site-packages/numpy/core/include -I/Users/pauliuskomskis/Desktop/env/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/vaexfast.cpp -o build/temp.macosx-10.9-x86_64-3.9/src/vaexfast.o -std=c++11 -mfpmath=sse -O3 -funroll-loops -g -mmacosx-version-min=10.9 In file included from src/vaexfast.cpp:31: In file included from /Users/pauliuskomskis/Desktop/env/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4: In file included from /Users/pauliuskomskis/Desktop/env/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12: In file included from /Users/pauliuskomskis/Desktop/env/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822: /Users/pauliuskomskis/Desktop/env/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]

warning "Using deprecated NumPy API, disable it with " \

 ^
src/vaexfast.cpp:743:2: warning: '/*' within block comment [-Wcomment]
        /**/
        ^
src/vaexfast.cpp:665:12: warning: unused variable 'i_x' [-Wunused-variable]
        long long i_x = offset_x;
                  ^
src/vaexfast.cpp:666:12: warning: unused variable 'i_y' [-Wunused-variable]
        long long i_y = offset_y;
                  ^
4 warnings generated.
g++ -bundle -undefined dynamic_lookup -arch x86_64 -g build/temp.macosx-10.9-x86_64-3.9/src/vaexfast.o -o build/lib.macosx-10.9-x86_64-3.9/vaex/vaexfast.cpython-39-darwin.so
building 'vaex.superstrings' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Users/pauliuskomskis/Desktop/env/lib/python3.9/site-packages/numpy/core/include -Ivendor/pybind11/include -Ivendor/pybind11/include -Ivendor/string-view-lite/include -Ivendor/boost -I/Users/pauliuskomskis/Desktop/env/include -I/Users/pauliuskomskis/Desktop/env/Library/include -I/private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/vendor/pcre/Library/include -I/Users/pauliuskomskis/Desktop/env/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/strings.cpp -o build/temp.macosx-10.9-x86_64-3.9/src/strings.o -std=c++11 -mfpmath=sse -O3 -funroll-loops -g -mmacosx-version-min=10.9
In file included from src/strings.cpp:7:
In file included from vendor/pybind11/include/pybind11/pybind11.h:44:
In file included from vendor/pybind11/include/pybind11/attr.h:13:
In file included from vendor/pybind11/include/pybind11/cast.h:16:
vendor/pybind11/include/pybind11/detail/internals.h:200:9: warning: 'PyEval_InitThreads' is deprecated [-Wdeprecated-declarations]
        PyEval_InitThreads();
        ^
/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/ceval.h:130:1: note: 'PyEval_InitThreads' has been explicitly marked deprecated here
Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
^
/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/pyport.h:508:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
In file included from src/strings.cpp:7:
In file included from vendor/pybind11/include/pybind11/pybind11.h:44:
In file included from vendor/pybind11/include/pybind11/attr.h:13:
vendor/pybind11/include/pybind11/cast.h:2007:26: warning: loop variable 'a' is always a copy because the range of type 'detail::args_proxy' does not return a reference [-Wrange-loop-analysis]
        for (const auto &a : ap)
                         ^
vendor/pybind11/include/pybind11/cast.h:2007:14: note: use non-reference type 'pybind11::handle'
        for (const auto &a : ap)
             ^~~~~~~~~~~~~~~
vendor/pybind11/include/pybind11/cast.h:2039:26: warning: loop variable 'k' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
        for (const auto &k : reinterpret_borrow<dict>(kp)) {
                         ^
vendor/pybind11/include/pybind11/cast.h:2039:14: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
        for (const auto &k : reinterpret_borrow<dict>(kp)) {
             ^~~~~~~~~~~~~~~
In file included from src/strings.cpp:7:
vendor/pybind11/include/pybind11/pybind11.h:1398:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries) {
                                 ^
vendor/pybind11/include/pybind11/pybind11.h:1398:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries) {
                     ^~~~~~~~~~~~~~~~
vendor/pybind11/include/pybind11/pybind11.h:1410:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries) {
                                 ^
vendor/pybind11/include/pybind11/pybind11.h:1410:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries) {
                     ^~~~~~~~~~~~~~~~
vendor/pybind11/include/pybind11/pybind11.h:1425:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries) {
                                 ^
vendor/pybind11/include/pybind11/pybind11.h:1425:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries) {
                     ^~~~~~~~~~~~~~~~
vendor/pybind11/include/pybind11/pybind11.h:1439:34: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
                for (const auto &kv : entries)
                                 ^
vendor/pybind11/include/pybind11/pybind11.h:1439:22: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
                for (const auto &kv : entries)
                     ^~~~~~~~~~~~~~~~
vendor/pybind11/include/pybind11/pybind11.h:1516:26: warning: loop variable 'kv' is always a copy because the range of type 'pybind11::dict' does not return a reference [-Wrange-loop-analysis]
        for (const auto &kv : entries)
                         ^
vendor/pybind11/include/pybind11/pybind11.h:1516:14: note: use non-reference type 'std::__1::pair<pybind11::handle, pybind11::handle>'
        for (const auto &kv : entries)
             ^~~~~~~~~~~~~~~~
src/strings.cpp:28:10: fatal error: 'pcrecpp.h' file not found
#include <pcrecpp.h>
         ^~~~~~~~~~~
8 warnings and 1 error generated.
error: command '/usr/bin/gcc' failed with exit code 1
----------------------------------------

ERROR: Command errored out with exit status 1: /Users/pauliuskomskis/Desktop/env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/setup.py'"'"'; file='"'"'/private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-install-obxqb5sd/vaex-core_94393d94518c4f1bb39d9f92313d5ba8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/0s/g0hqggvs5ygfvjvsbc5gdd740000gn/T/pip-record-_o4_m3uz/install-record.txt --single-version-externally-managed --compile --install-headers /Users/pauliuskomskis/Desktop/env/include/site/python3.9/vaex-core Check the logs for full command output.

JovanVeljanoski commented 3 years ago

I can see that your system is using cache so you are picking up old versions.. (from the logs you attached).

Maybe try fresh install, either through conda or pip?

You can try: pip install vaex==4.0.0a13 for the latest alpha via pip or conda install -c conda-forge vaex=3.0.0 for the last stable release.

JovanVeljanoski commented 3 years ago

Will close as stale. Please re-open if needed.