search

rapidsai / kvikio

KvikIO - High Performance File IO
https://docs.rapids.ai/api/kvikio/stable/
Apache License 2.0
148 stars 54 forks source link

Very Chatty CPU-only Build #444

Open Jacobfaib opened 3 weeks ago

Jacobfaib commented 3 weeks ago

When building KvikIO without CUDA, cmake spits out a great deal of scary looking but ultimately harmless warnings. Could these be silenced?

CMake Warning at arch-conda/cmake_build/_deps/kvikio-src/cpp/CMakeLists.txt:64 (message):
    Building KvikIO without cuFile

CMake Error at arch-conda/cmake_build/_deps/kvikio-src/cpp/cmake/thirdparty/get_nvtx.cmake:18 (include):
    include could not find requested file:

      $SRC_DIR/arch-conda/cmake_build/_deps/rapids-cmake-src/rapids-cmake/cpm/nvtx3.cmake
  Call Stack (most recent call first):
    arch-conda/cmake_build/_deps/kvikio-src/cpp/cmake/thirdparty/get_nvtx.cmake:25 (find_and_configure_nvtx)
    arch-conda/cmake_build/_deps/kvikio-src/cpp/CMakeLists.txt:83 (include)

  CMake Error at arch-conda/cmake_build/_deps/kvikio-src/cpp/cmake/thirdparty/get_thread_pool.cmake:18 (include):
    include could not find requested file:

      $SRC_DIR/arch-conda/cmake_build/_deps/rapids-cmake-src/rapids-cmake/cpm/bs_thread_pool.cmake
  Call Stack (most recent call first):
    arch-conda/cmake_build/_deps/kvikio-src/cpp/cmake/thirdparty/get_thread_pool.cmake:25 (find_and_configure_thread_pool)
    arch-conda/cmake_build/_deps/kvikio-src/cpp/CMakeLists.txt:84 (include)

 CMake Warning at arch-conda/cmake_build/_deps/kvikio-src/cpp/CMakeLists.txt:108 (message):
    Building KvikIO without CUDA

It seems that if CUDAToolKit is not found (or, perhaps the user passes an imaginary KvikIO_USE_CUDA=OFF), then KvikIO could:

  1. Not search for these packages in the first place, or...
  2. Search for them if it must, but error silently if they are not found
qkoziol commented 1 week ago

Agree, this is difficult to read and determine which failures are important.

madsbk commented 1 week ago

I think this is a genuine error:

CMake Error at arch-conda/cmake_build/_deps/kvikio-src/cpp/cmake/thirdparty/get_thread_pool.cmake:18 (include):
    include could not find requested file:

      $SRC_DIR/arch-conda/cmake_build/_deps/rapids-cmake-src/rapids-cmake/cpm/bs_thread_pool.cmake

I think it is because you need the rapids-build-backend conda package. This is my no-cuda env: 

$ conda list
# packages in environment at /home/mkristensen/apps/miniforge3/envs/kvikio-no-cuda-0804:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h69a702a_16    conda-forge
_sysroot_linux-aarch64_curr_repodata_hack 4                   h57d6b7b_16    conda-forge
alabaster                 0.7.16             pyhd8ed1ab_0    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
attrs                     24.2.0             pyh71513ae_0    conda-forge
aws-c-auth                0.7.26               hc36b679_2    conda-forge
aws-c-cal                 0.7.4                h2abdd08_0    conda-forge
aws-c-common              0.9.27               h4bc722e_0    conda-forge
aws-c-compression         0.2.19               haa50ccc_0    conda-forge
aws-c-event-stream        0.4.3                h570d160_0    conda-forge
aws-c-http                0.8.8                h9b61739_1    conda-forge
aws-c-io                  0.14.18              h49c7fd3_7    conda-forge
aws-c-mqtt                0.10.4              h5c8269d_18    conda-forge
aws-c-s3                  0.6.4               h77088c0_11    conda-forge
aws-c-sdkutils            0.1.19               h038f3f9_2    conda-forge
aws-checksums             0.1.18              h038f3f9_10    conda-forge
aws-crt-cpp               0.28.2               hf262114_0    conda-forge
aws-sdk-cpp               1.11.379             hc1bef60_8    conda-forge
azure-core-cpp            1.13.0               h935415a_0    conda-forge
azure-identity-cpp        1.8.0                hd126650_2    conda-forge
azure-storage-blobs-cpp   12.12.0              hd2e3451_0    conda-forge
azure-storage-common-cpp  12.7.0               h10ac4d7_1    conda-forge
azure-storage-files-datalake-cpp 12.11.0              h325d260_1    conda-forge
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
binutils                  2.40                 h4852527_7    conda-forge
binutils_impl_linux-64    2.40                 ha1999f0_7    conda-forge
binutils_impl_linux-aarch64 2.40                 hdca1da1_7    conda-forge
binutils_linux-64         2.40                 hb3c18ed_4    conda-forge
binutils_linux-aarch64    2.40                 h418c4e9_4    conda-forge
bokeh                     3.5.2              pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311hfdbb021_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.33.1               heb4867d_0    conda-forge
c-compiler                1.5.2                h0b41bf4_0    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
cffi                      1.17.0          py311hf29c0ef_1    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cloudpickle               3.0.0              pyhd8ed1ab_0    conda-forge
cmake                     3.30.3               hf9cb763_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.3.0           py311hd18a35c_1    conda-forge
coverage                  7.6.1           py311h9ecbd09_1    conda-forge
cxx-compiler              1.5.2                hf52228f_0    conda-forge
cython                    3.0.11          py311hfdbb021_2    conda-forge
cytoolz                   0.12.3          py311h459d7ec_0    conda-forge
dask                      2024.8.2           pyhd8ed1ab_0    conda-forge
dask-core                 2024.8.2           pyhd8ed1ab_0    conda-forge
dask-expr                 1.1.13             pyhd8ed1ab_0    conda-forge
distlib                   0.3.8              pyhd8ed1ab_0    conda-forge
distributed               2024.8.2           pyhd8ed1ab_0    conda-forge
docutils                  0.20.1          py311h38be061_3    conda-forge
doxygen                   1.9.1                hb166930_1    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
filelock                  3.15.4             pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2024.6.1           pyhff2d567_0    conda-forge
gcc                       11.4.0              h602e360_13    conda-forge
gcc_impl_linux-64         11.4.0              h00c12a0_13    conda-forge
gcc_impl_linux-aarch64    11.4.0              hdfcf976_13    conda-forge
gcc_linux-64              11.4.0               ha077dfb_4    conda-forge
gcc_linux-aarch64         11.4.0               h0bcf638_4    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.7.1                hbabe93e_0    conda-forge
gxx                       11.4.0              h602e360_13    conda-forge
gxx_impl_linux-64         11.4.0              h634f3ee_13    conda-forge
gxx_linux-64              11.4.0               h35bfe5d_4    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
identify                  2.6.0              pyhd8ed1ab_0    conda-forge
idna                      3.8                pyhd8ed1ab_0    conda-forge
imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        8.4.0              pyha770c72_0    conda-forge
importlib-resources       6.4.4              pyhd8ed1ab_0    conda-forge
importlib_metadata        8.4.0                hd8ed1ab_0    conda-forge
importlib_resources       6.4.4              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jsonschema                4.23.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
kernel-headers_linux-64   3.10.0              h4a8ded7_16    conda-forge
kernel-headers_linux-aarch64 4.18.0              h5b4a56d_16    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
ld_impl_linux-aarch64     2.40                 h4607a90_7    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20240116.2      cxx17_he02047a_1    conda-forge
libarrow                  17.0.0          h8d2e343_13_cpu    conda-forge
libarrow-acero            17.0.0          h5888daf_13_cpu    conda-forge
libarrow-dataset          17.0.0          h5888daf_13_cpu    conda-forge
libarrow-substrait        17.0.0          hf54134d_13_cpu    conda-forge
libblas                   3.9.0           23_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hb9d3cd8_2    conda-forge
libbrotlidec              1.1.0                hb9d3cd8_2    conda-forge
libbrotlienc              1.1.0                hb9d3cd8_2    conda-forge
libcblas                  3.9.0           23_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.9.1                hdb1bdb2_0    conda-forge
libdeflate                1.21                 h4bc722e_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.1.0               h77fa898_1    conda-forge
libgcc-devel_linux-64     11.4.0             h8f596e0_113    conda-forge
libgcc-devel_linux-aarch64 11.4.0              h44af0e8_13    conda-forge
libgcc-ng                 14.1.0               h69a702a_1    conda-forge
libgfortran               14.1.0               h69a702a_1    conda-forge
libgfortran-ng            14.1.0               h69a702a_1    conda-forge
libgfortran5              14.1.0               hc5f4f2c_1    conda-forge
libgomp                   14.1.0               h77fa898_1    conda-forge
libgoogle-cloud           2.28.0               h26d7fe4_0    conda-forge
libgoogle-cloud-storage   2.28.0               ha262f82_0    conda-forge
libgrpc                   1.62.2               h15f2491_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           23_linux64_openblas    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
libparquet                17.0.0          h39682fd_13_cpu    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libprotobuf               4.25.3               h08a7969_0    conda-forge
libre2-11                 2023.09.01           h5a48ba9_2    conda-forge
libsanitizer              11.4.0              h5763a12_13    conda-forge
libsqlite                 3.46.1               hadc24fc_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx                 14.1.0               hc0a3c3a_1    conda-forge
libstdcxx-devel_linux-64  11.4.0             h8f596e0_113    conda-forge
libstdcxx-ng              14.1.0               h4852527_1    conda-forge
libthrift                 0.20.0               h0e7cc3e_1    conda-forge
libtiff                   4.6.0                h46a8edc_4    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.48.0               hd590300_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.16                 hb9d3cd8_1    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.7               he7c6b58_4    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.3.3           py311h2cbdf9a_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markupsafe                2.1.5           py311h9ecbd09_1    conda-forge
msgpack-python            1.0.8           py311hd18a35c_1    conda-forge
ncurses                   6.5                  he02047a_1    conda-forge
ninja                     1.12.1               h297d8ca_0    conda-forge
nodeenv                   1.9.1              pyhd8ed1ab_0    conda-forge
numcodecs                 0.11.0          py311hcafe171_1    conda-forge
numpy                     2.1.1           py311h71ddf71_0    conda-forge
numpydoc                  1.8.0              pyhd8ed1ab_0    conda-forge
openjpeg                  2.5.2                h488ebb8_0    conda-forge
openssl                   3.3.2                hb9d3cd8_0    conda-forge
orc                       2.0.2                h669347b_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2           py311h14de704_1    conda-forge
partd                     1.4.2              pyhd8ed1ab_0    conda-forge
pathspec                  0.12.1             pyhd8ed1ab_0    conda-forge
pillow                    10.4.0          py311h82a398c_0    conda-forge
pip                       24.2               pyh8b19718_1    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
pre-commit                3.8.0              pyha770c72_0    conda-forge
psutil                    6.0.0           py311h331c9d8_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyarrow                   17.0.0          py311hbd00459_1    conda-forge
pyarrow-core              17.0.0          py311h4510849_1_cpu    conda-forge
pyarrow-hotfix            0.6                pyhd8ed1ab_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    8.3.2              pyhd8ed1ab_0    conda-forge
pytest-cov                5.0.0              pyhd8ed1ab_0    conda-forge
python                    3.11.9          hb806964_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    5_cp311    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.2           py311h9ecbd09_1    conda-forge
rapids-build-backend      0.3.2                      py_0    rapidsai
rapids-dependency-file-generator 1.14.0                     py_0    rapidsai
re2                       2023.09.01           h7f4b329_2    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.35.1             pyhd8ed1ab_0    conda-forge
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
rhash                     1.4.4                hd590300_0    conda-forge
rpds-py                   0.20.0          py311h9e33e62_1    conda-forge
s2n                       1.5.1                h3400bea_0    conda-forge
scikit-build-core         0.10.5             pyh4afc917_0    conda-forge
setuptools                73.0.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.2.1                ha2e4443_0    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sphinx                    7.4.7              pyhd8ed1ab_0    conda-forge
sphinx-click              6.0.0              pyhd8ed1ab_0    conda-forge
sphinx_rtd_theme          2.0.0              pyha770c72_0    conda-forge
sphinxcontrib-applehelp   2.0.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     2.0.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-htmlhelp    2.1.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jquery      4.1                pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      2.0.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-serializinghtml 1.1.10             pyhd8ed1ab_0    conda-forge
sysroot_linux-64          2.17                h4a8ded7_16    conda-forge
sysroot_linux-aarch64     2.17                h5b4a56d_16    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tblib                     3.0.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.13.2             pyha770c72_0    conda-forge
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
tornado                   6.4.1           py311h9ecbd09_1    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tzdata                    2024a                h8827d51_1    conda-forge
ukkonen                   1.0.1           py311h9547e67_4    conda-forge
urllib3                   2.2.2              pyhd8ed1ab_1    conda-forge
virtualenv                20.26.3            pyhd8ed1ab_0    conda-forge
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xyzservices               2024.9.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zarr                      2.18.2             pyhd8ed1ab_0    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.20.1             pyhd8ed1ab_0    conda-forge
zstandard                 0.23.0          py311hbc35293_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge
madsbk commented 1 week ago

Can you check if https://github.com/rapidsai/kvikio/pull/455 solves anything?