rapidsai / cudf

cuDF - GPU DataFrame Library
https://docs.rapids.ai/api/cudf/stable/
Apache License 2.0
8.23k stars 883 forks source link

[BUG]cudf branch V22.12 build cudf python package error at cython compiling #12731

Closed tianyuanzhangNB closed 1 year ago

tianyuanzhangNB commented 1 year ago

Describe the bug

:cudf$ ./build.sh cudf
Compiling cudf/_lib/avro.pyx because it changed.
Compiling cudf/_lib/binaryop.pyx because it changed.
Compiling cudf/_lib/column.pyx because it changed.
Compiling cudf/_lib/concat.pyx because it changed.
Compiling cudf/_lib/copying.pyx because it changed.
Compiling cudf/_lib/csv.pyx because it changed.
Compiling cudf/_lib/datetime.pyx because it changed.
Compiling cudf/_lib/filling.pyx because it changed.
Compiling cudf/_lib/gpuarrow.pyx because it changed.
Compiling cudf/_lib/groupby.pyx because it changed.
Compiling cudf/_lib/hash.pyx because it changed.
Compiling cudf/_lib/interop.pyx because it changed.
Compiling cudf/_lib/io/datasource.pyx because it changed.
Compiling cudf/_lib/io/utils.pyx because it changed.
Compiling cudf/_lib/join.pyx because it changed.
Compiling cudf/_lib/json.pyx because it changed.
Compiling cudf/_lib/labeling.pyx because it changed.
Compiling cudf/_lib/lists.pyx because it changed.
Compiling cudf/_lib/merge.pyx because it changed.
Compiling cudf/_lib/null_mask.pyx because it changed.
Compiling cudf/_lib/nvtext/edit_distance.pyx because it changed.
Compiling cudf/_lib/nvtext/generate_ngrams.pyx because it changed.
Compiling cudf/_lib/nvtext/ngrams_tokenize.pyx because it changed.
Compiling cudf/_lib/nvtext/normalize.pyx because it changed.
Compiling cudf/_lib/nvtext/replace.pyx because it changed.
Compiling cudf/_lib/nvtext/stemmer.pyx because it changed.
Compiling cudf/_lib/nvtext/subword_tokenize.pyx because it changed.
Compiling cudf/_lib/nvtext/tokenize.pyx because it changed.
Compiling cudf/_lib/orc.pyx because it changed.
Compiling cudf/_lib/parquet.pyx because it changed.
Compiling cudf/_lib/partitioning.pyx because it changed.
Compiling cudf/_lib/quantiles.pyx because it changed.
Compiling cudf/_lib/reduce.pyx because it changed.
Compiling cudf/_lib/replace.pyx because it changed.
Compiling cudf/_lib/reshape.pyx because it changed.
Compiling cudf/_lib/rolling.pyx because it changed.
Compiling cudf/_lib/round.pyx because it changed.
Compiling cudf/_lib/scalar.pyx because it changed.
Compiling cudf/_lib/search.pyx because it changed.
Compiling cudf/_lib/sort.pyx because it changed.
Compiling cudf/_lib/stream_compaction.pyx because it changed.
Compiling cudf/_lib/string_casting.pyx because it changed.
Compiling cudf/_lib/strings/attributes.pyx because it changed.
Compiling cudf/_lib/strings/capitalize.pyx because it changed.
Compiling cudf/_lib/strings/case.pyx because it changed.
Compiling cudf/_lib/strings/char_types.pyx because it changed.
Compiling cudf/_lib/strings/combine.pyx because it changed.
Compiling cudf/_lib/strings/contains.pyx because it changed.
Compiling cudf/_lib/strings/convert/convert_fixed_point.pyx because it changed.
Compiling cudf/_lib/strings/convert/convert_floats.pyx because it changed.
Compiling cudf/_lib/strings/convert/convert_integers.pyx because it changed.
Compiling cudf/_lib/strings/convert/convert_urls.pyx because it changed.
Compiling cudf/_lib/strings/extract.pyx because it changed.
Compiling cudf/_lib/strings/find.pyx because it changed.
Compiling cudf/_lib/strings/find_multiple.pyx because it changed.
Compiling cudf/_lib/strings/findall.pyx because it changed.
Compiling cudf/_lib/strings/padding.pyx because it changed.
Compiling cudf/_lib/strings/replace.pyx because it changed.
Compiling cudf/_lib/strings/replace_re.pyx because it changed.
Compiling cudf/_lib/strings/split/partition.pyx because it changed.
Compiling cudf/_lib/strings/split/split.pyx because it changed.
Compiling cudf/_lib/strings/strip.pyx because it changed.
Compiling cudf/_lib/strings/substring.pyx because it changed.
Compiling cudf/_lib/strings/translate.pyx because it changed.
Compiling cudf/_lib/strings/wrap.pyx because it changed.
Compiling cudf/_lib/table.pyx because it changed.
Compiling cudf/_lib/transform.pyx because it changed.
Compiling cudf/_lib/transpose.pyx because it changed.
Compiling cudf/_lib/unary.pyx because it changed.
Compiling cudf/_lib/utils.pyx because it changed.
[ 1/70] Cythonizing cudf/_lib/avro.pyx
warning: cudf/_lib/cpp/types.pxd:51:8: 'UNEQUAL' redeclared 

Error compiling Cython file:
------------------------------------------------------------
...

from libcpp.memory cimport unique_ptr
from libcpp.vector cimport vector
from libcpp cimport bool

from rmm._lib.device_buffer cimport device_buffer
^
------------------------------------------------------------

cudf/_lib/cpp/column/column.pxd:7:0: 'rmm/_lib/device_buffer.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...

from libcpp.memory cimport unique_ptr
from libcpp.vector cimport vector
from libcpp cimport bool

from rmm._lib.device_buffer cimport device_buffer
^
------------------------------------------------------------

cudf/_lib/cpp/column/column.pxd:7:0: 'rmm/_lib/device_buffer/device_buffer.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
    column_view, mutable_column_view
)

cdef extern from "cudf/column/column.hpp" namespace "cudf" nogil:
    cdef cppclass column_contents "cudf::column::contents":
        unique_ptr[device_buffer] data
                               ^

Steps/Code to reproduce bug after ./build.sh libcudf run ./build.sh cudf get the error

Expected behavior build cudf python package pass

Environment overview (please complete the following information)

Environment details :cudf$ ./print_env.sh

Click here to see environment details

     **git***
     commit deee1f62d1f1887b625ddd6b8a7cbd393f04c264 (HEAD -> branch-0.19, origin/branch-0.19)
     Author: AJ Schmidt 
     Date:   Thu May 20 15:52:28 2021 -0400

     update changelog (#8297)
     **git submodules***

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=20.04
     DISTRIB_CODENAME=focal
     DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
     NAME="Ubuntu"
     VERSION="20.04.3 LTS (Focal Fossa)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 20.04.3 LTS"
     VERSION_ID="20.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=focal
     UBUNTU_CODENAME=focal
     Linux iluvatar-ThinkBook-14-G3-ITL 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Wed Feb  8 16:54:55 2023
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 450.216.04   Driver Version: 450.216.04   CUDA Version: 11.0     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  GeForce MX450       Off  | 00000000:2C:00.0 Off |                  N/A |
     | N/A   38C    P0    N/A /  N/A |     10MiB /  1878MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |    0   N/A  N/A      1020      G   /usr/lib/xorg/Xorg                  4MiB |
     |    0   N/A  N/A      1772      G   /usr/lib/xorg/Xorg                  4MiB |
     +-----------------------------------------------------------------------------+

     ***CPU***
     架构:                           x86_64
     CPU 运行模式:                   32-bit, 64-bit
     字节序:                         Little Endian
     Address sizes:                   39 bits physical, 48 bits virtual
     CPU:                             8
     在线 CPU 列表:                  0-7
     每个核的线程数:                 2
     每个座的核数:                   4
     座:                             1
     NUMA 节点:                      1
     厂商 ID:                        GenuineIntel
     CPU 系列:                       6
     型号:                           140
     型号名称:                       11th Gen Intel(R) Core(TM) i7-1195G7 @ 2.90GHz
     步进:                           2
     CPU MHz:                        2900.000
     CPU 最大 MHz:                   5000.0000
     CPU 最小 MHz:                   400.0000
     BogoMIPS:                       5836.80
     虚拟化:                         VT-x
     L1d 缓存:                       192 KiB
     L1i 缓存:                       128 KiB
     L2 缓存:                        5 MiB
     L3 缓存:                        12 MiB
     NUMA 节点0 CPU:                 0-7
     Vulnerability Itlb multihit:     Not affected
     Vulnerability L1tf:              Not affected
     Vulnerability Mds:               Not affected
     Vulnerability Meltdown:          Not affected
     Vulnerability Mmio stale data:   Not affected
     Vulnerability Retbleed:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Not affected
     标记:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear flush_l1d arch_capabilities

     ***CMake***
     /usr/local/bin/cmake
     cmake version 3.24.3

     CMake suite maintained and supported by Kitware (kitware.com/cmake).

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0
     Copyright (C) 2020 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***
     /usr/local/cuda/bin/nvcc
     nvcc: NVIDIA (R) Cuda compiler driver
     Copyright (c) 2005-2021 NVIDIA Corporation
     Built on Mon_Sep_13_19:13:29_PDT_2021
     Cuda compilation tools, release 11.5, V11.5.50
     Build cuda_11.5.r11.5/compiler.30411180_0

     ***Python***
     /usr/bin/python
     Python 3.8.10

     ***Environment Variables***
     PATH                            : /usr/local/cuda/bin:/home/iluvatar/miniconda3/condabin:/opt/ros/noetic/bin:/home/iluvatar/.cargo/bin:/home/iluvatar/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
     LD_LIBRARY_PATH                 : /usr/local/cuda/lib64:/opt/ros/noetic/lib
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    :
     PYTHON_PATH                     :

     ***conda packages***
     /home/iluvatar/miniconda3/condabin/conda
     # packages in environment at /home/iluvatar/miniconda3:
     #
     # Name                    Version                   Build  Channel
     _libgcc_mutex             0.1                        main
     _openmp_mutex             5.1                       1_gnu
     brotlipy                  0.7.0           py310h7f8727e_1002
     bzip2                     1.0.8                h7b6447c_0
     ca-certificates           2022.10.11           h06a4308_0
     certifi                   2022.12.7       py310h06a4308_0
     cffi                      1.15.1          py310h5eee18b_3
     charset-normalizer        2.0.4              pyhd3eb1b0_0
     conda                     22.11.1         py310h06a4308_4
     conda-content-trust       0.1.3           py310h06a4308_0
     conda-package-handling    1.9.0           py310h5eee18b_1
     cryptography              38.0.1          py310h9ce1e76_0
     idna                      3.4             py310h06a4308_0
     ld_impl_linux-64          2.38                 h1181459_1
     libffi                    3.4.2                h6a678d5_6
     libgcc-ng                 11.2.0               h1234567_1
     libgomp                   11.2.0               h1234567_1
     libstdcxx-ng              11.2.0               h1234567_1
     libuuid                   1.41.5               h5eee18b_0
     ncurses                   6.3                  h5eee18b_3
     openssl                   1.1.1s               h7f8727e_0
     pip                       22.3.1          py310h06a4308_0
     pluggy                    1.0.0           py310h06a4308_1
     pycosat                   0.6.4           py310h5eee18b_0
     pycparser                 2.21               pyhd3eb1b0_0
     pyopenssl                 22.0.0             pyhd3eb1b0_0
     pysocks                   1.7.1           py310h06a4308_0
     python                    3.10.8               h7a1cb2a_1
     readline                  8.2                  h5eee18b_0
     requests                  2.28.1          py310h06a4308_0
     ruamel.yaml               0.17.21         py310h5eee18b_0
     ruamel.yaml.clib          0.2.6           py310h5eee18b_1
     setuptools                65.5.0          py310h06a4308_0
     six                       1.16.0             pyhd3eb1b0_1
     sqlite                    3.40.0               h5082296_0
     tk                        8.6.12               h1ccaba5_0
     toolz                     0.12.0          py310h06a4308_0
     tqdm                      4.64.1          py310h06a4308_0
     tzdata                    2022g                h04d1e81_0
     urllib3                   1.26.13         py310h06a4308_0
     wheel                     0.37.1             pyhd3eb1b0_0
     xz                        5.2.8                h5eee18b_0
     zlib                      1.2.13               h5eee18b_0

Additional context Add any other context about the problem here.

bdice commented 1 year ago

@tianyuanzhangNB Do you have the rmm Python package installed? If so, how did you install it?

GregoryKimball commented 1 year ago

Hello @tianyuanzhangNB please check out our current branch 23.08 and let us know if you have any trouble building cuDF-python. I recommend using our nightly docker containers as the fastest way to get a working build.