search

david-cortes / isotree

(Python, R, C/C++) Isolation Forest and variations such as SCiForest and EIF, with some additions (outlier detection + similarity + NA imputation)
https://isotree.readthedocs.io
BSD 2-Clause "Simplified" License
186 stars 38 forks source link

Isotree==0.5.22 is failing on databricks cluster #55

Closed Ravikumarnl closed 1 year ago

Ravikumarnl commented 1 year ago

Hi,

I am trying to install isotree 0.5.22 on databricks cluster (12.2 LTS ML (includes Apache Spark 3.3.2, Scala 2.12)), however it is giving issues while installing it (even after upgrading pip to latest version: 23.2.1, similar error appears)

  Error compiling Cython file:   ------------------------------------------------------------   ...       void tmat_to_dense(double tmat, double dmat, size_t n, double fill_diag)         void merge_models(IsoForest     model,      IsoForest     other,                         ExtIsoForest  ext_model,  ExtIsoForest  ext_other,                         Imputer       imputer,    Imputer       iother,                         TreesIndexer  indexer,    TreesIndexer  ind_other) except + nogil                                                                                           ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:223:89: undeclared name not builtin: nogil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil  

completer Error message: image

Ravikumarnl commented 1 year ago

Complete error message is as below: (not copied completely in previous post)

Library installation attempted on the driver node of cluster 0714-111409-o9fj6ac2 and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: org.apache.spark.SparkException: Process List(bash, /local_disk0/.ephemeral_nfs/cluster_libraries/python/python_start_clusterwide.sh, /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/pip, install, isotree==0.5.22, --disable-pip-version-check) exited with code 1.   ERROR: Command errored out with exit status 1:    command: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python /local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpz3gi_800        cwd: /tmp/pip-install-vnnaag20/isotree_5156cd302e594e3b88594b6810c157c7   Complete output (574 lines):   /databricks/python/lib/python3.9/site-packages/setuptools/_distutils/extension.py:131: UserWarning: Unknown Extension options: 'install_requires'     warnings.warn(msg)   running bdist_wheel   running build   running build_py   creating build   creating build/lib.linux-x86_64-3.9   creating build/lib.linux-x86_64-3.9/isotree   copying isotree/init.py -> build/lib.linux-x86_64-3.9/isotree   running build_ext   --- Checking compiler support for option '-march=native'   --- Checking compiler support for option '-fopenmp'   --- Checking compiler support for '__restrict' qualifier   --- Checking compiler support for option '-O3'   --- Checking compiler support for option '-fno-math-errno'   --- Checking compiler support for option '-fno-trapping-math'   --- Checking compiler support for option '-std=c++17'   --- Checking compiler support for option '-flto=auto'   cythoning isotree/cpp_interface.pyx to isotree/cpp_interface.cpp     Error compiling Cython file:   ------------------------------------------------------------   ...       void tmat_to_dense(double tmat, double dmat, size_t n, double fill_diag)         void merge_models(IsoForest     model,      IsoForest     other,                         ExtIsoForest  ext_model,  ExtIsoForest  ext_other,                         Imputer       imputer,    Imputer       iother,                         TreesIndexer  indexer,    TreesIndexer  ind_other) except + nogil                                                                                           ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:223:89: undeclared name not builtin: nogil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...             cdef int ret_val = 0             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               ret_val = \               fit_iforest(model_ptr, ext_model_ptr,                         ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:823:23: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               add_tree(model_ptr, ext_model_ptr,                      ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1024:20: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               add_tree(model_ptr, ext_model_ptr,                      ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1024:20: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               add_tree(model_ptr, ext_model_ptr,                      ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1024:20: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               add_tree(model_ptr, ext_model_ptr,                      ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1024:20: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               add_tree(model_ptr, ext_model_ptr,                      ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1024:20: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               add_tree(model_ptr, ext_model_ptr,                      ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1024:20: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               predict_iforest(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1164:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               predict_iforest(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1164:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               predict_iforest(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1164:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               predict_iforest(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1164:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               predict_iforest(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1164:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               predict_iforest(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1164:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               calc_similarity(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1288:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               calc_similarity(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1288:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               calc_similarity(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1288:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               calc_similarity(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1288:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               calc_similarity(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1288:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           cdef TreesIndexer  indexer_ptr = NULL           if not self.indexer.indices.empty():               indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               calc_similarity(numeric_data_ptr, categ_data_ptr,                             ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1288:27: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               model_ptr      =  &self.isoforest           else:               ext_model_ptr  =  &self.ext_isoforest             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               impute_missing_values(numeric_data_ptr, categ_data_ptr, is_col_major,                                   ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1374:33: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               model_ptr      =  &self.isoforest           else:               ext_model_ptr  =  &self.ext_isoforest             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               impute_missing_values(numeric_data_ptr, categ_data_ptr, is_col_major,                                   ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1374:33: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               model_ptr      =  &self.isoforest           else:               ext_model_ptr  =  &self.ext_isoforest             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               impute_missing_values(numeric_data_ptr, categ_data_ptr, is_col_major,                                   ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1374:33: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               model_ptr      =  &self.isoforest           else:               ext_model_ptr  =  &self.ext_isoforest             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               impute_missing_values(numeric_data_ptr, categ_data_ptr, is_col_major,                                   ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1374:33: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               model_ptr      =  &self.isoforest           else:               ext_model_ptr  =  &self.ext_isoforest             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               impute_missing_values(numeric_data_ptr, categ_data_ptr, is_col_major,                                   ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1374:33: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               model_ptr      =  &self.isoforest           else:               ext_model_ptr  =  &self.ext_isoforest             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               impute_missing_values(numeric_data_ptr, categ_data_ptr, is_col_major,                                   ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1374:33: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...               ptr_indexer = &self.indexer           if not other.indexer.indices.empty():               prt_ind_other = &other.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               merge_models(ptr_model, ptr_other,                          ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1424:24: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest             cdef vector[cpp_string] res           with nogil, boundscheck(False), nonecheck(False), wraparound(False):               res = generate_sql(model_ptr, ext_model_ptr,                                ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1546:30: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest           cdef TreesIndexer  indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               set_reference_points(model_ptr, ext_model_ptr, indexer_ptr,                                  ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1687:32: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest           cdef TreesIndexer  indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               set_reference_points(model_ptr, ext_model_ptr, indexer_ptr,                                  ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1687:32: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest           cdef TreesIndexer  indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               set_reference_points(model_ptr, ext_model_ptr, indexer_ptr,                                  ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1687:32: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest           cdef TreesIndexer  indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               set_reference_points(model_ptr, ext_model_ptr, indexer_ptr,                                  ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1687:32: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest           cdef TreesIndexer  indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               set_reference_points(model_ptr, ext_model_ptr, indexer_ptr,                                  ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1687:32: Calling gil-requiring function not allowed without gil     Error compiling Cython file:   ------------------------------------------------------------   ...           else:               ext_model_ptr  =  &self.ext_isoforest           cdef TreesIndexer  indexer_ptr = &self.indexer             with nogil, boundscheck(False), nonecheck(False), wraparound(False):               set_reference_points(model_ptr, ext_model_ptr, indexer_ptr,                                  ^   ------------------------------------------------------------     isotree/cpp_interface.pyx:1687:32: Calling gil-requiring function not allowed without gil   building 'isotree._cpp_interface' extension   creating build/temp.linux-x86_64-3.9   creating build/temp.linux-x86_64-3.9/isotree   creating build/temp.linux-x86_64-3.9/src   x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -D_USE_XOSHIRO -DNDEBUG -D_FOR_PYTHON "-DCYTHON_EXTERN_C=extern \"C\"" -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D_USE_ROBIN_MAP -I/databricks/python/lib/python3.9/site-packages/numpy/core/include -I. -I./src -I/local_disk0/.ephemeral_nfs/cluster_libraries/python/include -I/usr/include/python3.9 -c isotree/cpp_interface.cpp -o build/temp.linux-x86_64-3.9/isotree/cpp_interface.o -march=native -fopenmp -O3 -fno-math-errno -fno-trapping-math -std=c++17 -flto=auto   isotree/cpp_interface.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.       1 | #error Do not use this file, it is the result of a failed Cython compilation.         |  ^~~~~   error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1   ----------------------------------------   ERROR: Failed building wheel for isotree ERROR: Could not build wheels for isotree which use PEP 517 and cannot be installed directly Done

david-cortes commented 1 year ago

As outlined in the toml file, this package requires cython>=3.0.0, so you if are using something like PIP_CONSTRAINTS or some other system built on top of pip that makes it ignore such build nuances, it's up to you to make it satisfy those requirements.

If you didn't set up those things then maybe contact your provider's support as it suggests in the error message.

Ravikumarnl commented 1 year ago

Thanks David, It worked with cython==3.0.0