conda-forge / c-blosc2-feedstock

A conda-smithy repository for c-blosc2.
BSD 3-Clause "New" or "Revised" License
0 stars 10 forks source link

Cmake `find_package(blosc2)` not working #57

Open vicentebolea opened 6 months ago

vicentebolea commented 6 months ago

Solution to issue cannot be found in the documentation.

Issue

In Linux and windows

conda install conda-forge::c-blosc2
echo "find_package(blosc2)" > CMakeLists.txt
cmake .

Output

-- No LZ4 library found.  Using internal sources.
CMake Warning at cmake/DetectOptions.cmake:77 (find_package):
  Found package configuration file:

    /home/vicente/.miniconda3/lib/cmake/Blosc2/Blosc2Config.cmake

  but it set Blosc2_FOUND to FALSE so package "Blosc2" is considered to be
  NOT FOUND.  Reason given by package:

  Blosc2 could not be found because dependency LZ4 could not be found.

Call Stack (most recent call first):
  CMakeLists.txt:194 (include)

We have two solutions here:

  1. Build the c-blosc2 package with internal lz4
  2. Patch the findlz4.cmake in c-blosc2 to help it find lz4 installed in conda.

Installed packages

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
autopep8                  2.0.2                    pypi_0    pypi
boltons                   23.1.1             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311hb755f60_1    conda-forge
build                     0.10.0                   pypi_0    pypi
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.27.0               hd590300_0    conda-forge
c-blosc2                  2.13.2               hb4ffafa_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311hb3a22ac_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.6                    pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     24.1.2          py311h38be061_0    conda-forge
conda-content-trust       0.2.0              pyhd8ed1ab_0    conda-forge
conda-libmamba-solver     24.1.0             pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
cryptography              42.0.5          py311h63ff55d_0    conda-forge
distro                    1.9.0              pyhd8ed1ab_0    conda-forge
fmt                       10.2.1               h00ab1b0_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2                    pypi_0    pypi
jinja2-cli                0.8.2                    pypi_0    pypi
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py311h38be061_3    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.7.2                h2aa1ff5_1    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libmamba                  1.5.7                had39da4_0    conda-forge
libmambapy                1.5.7           py311hf2555c7_0    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libsolv                   0.7.28               hfc55251_0    conda-forge
libsqlite                 3.45.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.5               h232c23b_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
markupsafe                2.1.3                    pypi_0    pypi
menuinst                  2.0.2           py311h38be061_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
packaging                 24.0               pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pip-tools                 7.2.0                    pypi_0    pypi
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.4.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycodestyle               2.11.0                   pypi_0    pypi
pycosat                   0.6.6           py311h459d7ec_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyproject-hooks           1.0.0                    pypi_0    pypi
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.8          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4.post0         hd590300_1    conda-forge
reproc-cpp                14.2.4.post0         h59595ed_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.18.6          py311h459d7ec_0    conda-forge
ruamel.yaml.clib          0.2.8           py311h459d7ec_0    conda-forge
setuptools                69.2.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tqdm                      4.66.2             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zlib-ng                   2.0.7                h0b41bf4_0    conda-forge
zstandard                 0.22.0          py311haa97af0_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Environment info

→ conda info

     active environment : base
    active env location : /home/vicente/.miniconda3
            shell level : 1
       user config file : /home/vicente/.condarc
 populated config files : /home/vicente/.condarc
          conda version : 24.1.2
    conda-build version : not installed
         python version : 3.11.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.1.2=0
                          __cuda=12.4=0
                          __glibc=2.31=0
                          __linux=5.14.21=0
                          __unix=0=0
       base environment : /home/vicente/.miniconda3  (writable)
      conda av data dir : /home/vicente/.miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/vicente/.miniconda3/pkgs
                          /home/vicente/.conda/pkgs
       envs directories : /home/vicente/.miniconda3/envs
                          /home/vicente/.conda/envs
               platform : linux-64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.11.8 Linux/5.14.21-150500.55.49-default opensuse/15.5 glibc/2.31 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.7
                UID:GID : 1000:100
             netrc file : None
           offline mode : False
vicentebolea commented 6 months ago

@hmaarrfk

hmaarrfk commented 6 months ago

Thanks!

Can you please provide a cmake file and c file that you are trying to compile.

My guess looking at the upstream repo was that the CMakeFiles might have been a secondary concern. I only learned the standard mistakes after years....

hmaarrfk commented 6 months ago

sorry, i think i have my own code. but i'm having a hard time.

The first thing i'm noticing is that you didn't create a conda environment:

    /home/vicente/.miniconda3/lib/cmake/Blosc2/Blosc2Config.cmake

it is finding the base enviroment or smoething 's Blosc2Config.cmake.

you are also starting from miniconda3, so I'm worried there are some transition issues between Anaconda -> conda-forge.

Can you create an environment with:

mamba create --name dev --channel conda-forge --override-channels c-blosc2 cmake compilers
hmaarrfk commented 6 months ago
mamba create --name dev --channel conda-forge --override-channels c-blosc2 cmake compilers make
wget https://raw.githubusercontent.com/Blosc/c-blosc2/main/examples/simple.c
cat > CMakeLists.txt << EOF
cmake_minimum_required(VERSION 3.12)
project(ExampleBlosc2)

# Find the blosc2 library
find_package(Blosc2 REQUIRED)

# Define the executable
add_executable(simple simple.c)

# Link the blosc2 library
target_link_libraries(simple PRIVATE blosc2)
EOF

mamba activate dev
mkdir build
cd build
cmake ${CMAKE_ARGS} ..
make
seems to work.... ``` 21:15 $ mamba create --name dev --channel conda-forge --override-channels c-blosc2 cmake compilers make wget https://raw.githubusercontent.com/Blosc/c-blosc2/main/examples/simple.c cat > CMakeLists.txt << EOF cmake_minimum_required(VERSION 3.12) project(ExampleBlosc2) # Find the blosc2 library find_package(Blosc2 REQUIRED) # Define the executable add_executable(simple simple.c) # Link the blosc2 library target_link_libraries(simple PRIVATE blosc2) EOF mamba activate dev mkdir build cd build cmake ${CMAKE_ARGS} .. make Looking for: ['c-blosc2', 'cmake', 'compilers', 'make'] conda-forge/linux-64 Using cache conda-forge/noarch Using cache Transaction Prefix: /home/mark/miniforge3/envs/dev Updating specs: - c-blosc2 - cmake - compilers - make Package Version Build Channel Size ────────────────────────────────────────────────────────────────────────────────── Install: ────────────────────────────────────────────────────────────────────────────────── + _libgcc_mutex 0.1 conda_forge conda-forge Cached + libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge Cached + ld_impl_linux-64 2.40 h41732ed_0 conda-forge Cached + ca-certificates 2024.2.2 hbcca054_0 conda-forge Cached + libgomp 13.2.0 h807b86a_5 conda-forge Cached + _openmp_mutex 4.5 2_gnu conda-forge Cached + libgcc-ng 13.2.0 h807b86a_5 conda-forge Cached + ncurses 6.4 h59595ed_2 conda-forge Cached + libuv 1.48.0 hd590300_0 conda-forge Cached + libexpat 2.6.2 h59595ed_0 conda-forge Cached + bzip2 1.0.8 hd590300_5 conda-forge Cached + libgfortran5 13.2.0 ha4646dd_5 conda-forge Cached + libsanitizer 12.3.0 h0f45ef3_5 conda-forge Cached + c-ares 1.27.0 hd590300_0 conda-forge Cached + libev 4.33 hd590300_2 conda-forge Cached + lz4-c 1.9.4 hcb278e6_0 conda-forge Cached + openssl 3.2.1 hd590300_0 conda-forge Cached + libzlib 1.2.13 hd590300_5 conda-forge Cached + make 4.3 hd18ef5c_1 conda-forge Cached + keyutils 1.6.1 h166bdaf_0 conda-forge Cached + rhash 1.4.4 hd590300_0 conda-forge Cached + xz 5.2.6 h166bdaf_0 conda-forge Cached + zlib-ng 2.0.7 h0b41bf4_0 conda-forge Cached + libedit 3.1.20191231 he28a2e2_2 conda-forge Cached + libnghttp2 1.58.0 h47da74e_1 conda-forge Cached + libssh2 1.11.0 h0841786_0 conda-forge Cached + zstd 1.5.5 hfc55251_0 conda-forge Cached + krb5 1.21.2 h659d440_0 conda-forge Cached + c-blosc2 2.13.2 hb4ffafa_0 conda-forge Cached + libcurl 8.5.0 hca28451_0 conda-forge Cached + cmake 3.28.3 hcfe8598_0 conda-forge Cached + kernel-headers_linux-64 2.6.32 he073ed8_17 conda-forge Cached + libstdcxx-devel_linux-64 12.3.0 h8bca6fd_105 conda-forge Cached + libgcc-devel_linux-64 12.3.0 h8bca6fd_105 conda-forge Cached + sysroot_linux-64 2.12 he073ed8_17 conda-forge Cached + binutils_impl_linux-64 2.40 hf600244_0 conda-forge Cached + gcc_impl_linux-64 12.3.0 he2b93b0_5 conda-forge Cached + binutils_linux-64 2.40 hdade7a5_3 conda-forge Cached + binutils 2.40 hdd6e379_0 conda-forge Cached + gcc 12.3.0 h95e488c_3 conda-forge Cached + gfortran_impl_linux-64 12.3.0 hfcedea8_5 conda-forge Cached + gxx_impl_linux-64 12.3.0 he2b93b0_5 conda-forge Cached + gcc_linux-64 12.3.0 h6477408_3 conda-forge Cached + gfortran 12.3.0 h7389182_3 conda-forge Cached + gxx 12.3.0 h95e488c_3 conda-forge Cached + gxx_linux-64 12.3.0 h4a1b8e8_3 conda-forge Cached + gfortran_linux-64 12.3.0 h617cb40_3 conda-forge Cached + c-compiler 1.7.0 hd590300_0 conda-forge Cached + fortran-compiler 1.7.0 heb67821_0 conda-forge Cached + cxx-compiler 1.7.0 h00ab1b0_0 conda-forge Cached + compilers 1.7.0 ha770c72_0 conda-forge Cached Summary: Install: 51 packages Total download: 0 B ────────────────────────────────────────────────────────────────────────────────── Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done To activate this environment, use $ mamba activate dev To deactivate an active environment, use $ mamba deactivate --2024-03-15 21:15:16-- https://raw.githubusercontent.com/Blosc/c-blosc2/main/examples/simple.c Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.108.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2555 (2.5K) [text/plain] Saving to: ‘simple.c’ simple.c 100%[========================================================================================================================================================================================================================================================================================>] 2.50K --.-KB/s in 0s 2024-03-15 21:15:16 (9.89 MB/s) - ‘simple.c’ saved [2555/2555] -- The C compiler identification is GNU 12.3.0 -- The CXX compiler identification is GNU 12.3.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /home/mark/miniforge3/envs/dev/bin/x86_64-conda-linux-gnu-cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /home/mark/miniforge3/envs/dev/bin/x86_64-conda-linux-gnu-c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Check if compiler accepts -pthread -- Check if compiler accepts -pthread - yes -- Found Threads: TRUE -- Found LZ4 library: /home/mark/miniforge3/envs/dev/lib/liblz4.so -- Found ZLIB_NG: /home/mark/miniforge3/envs/dev/lib/libz-ng.so -- Found zlib-ng: /home/mark/miniforge3/envs/dev/lib/libz-ng.so, /home/mark/miniforge3/envs/dev/include -- Configuring done (0.3s) -- Generating done (0.0s) -- Build files have been written to: /home/mark/Downloads/blosc2/build [ 50%] Building C object CMakeFiles/simple.dir/simple.c.o [100%] Linking C executable simple [100%] Built target simple (dev) ✔ ~/Downloads/blosc2/build 21:15 $ ./simple Blosc version info: 2.13.2 ($Date:: 2023-02-07 #$) Compression: 40000000 -> 637877 (62.7x) Correctly extracted 5 elements from compressed chunk! Decompression successful! Successful roundtrip! ```
hmaarrfk commented 6 months ago

Note that I had to use Blosc2 and not blosc2

vicentebolea commented 6 months ago

@hmaarrfk thanks for the help. Few things:

I am sort of new to conda, is it mamba much faster?

I will close this as I can see that conda expects cmake users to use the cmake provided by conda. Otherwise, we have to set the CMAKE_PREFIX_PATH properly.

hmaarrfk commented 6 months ago

I will close this as I can see that conda expects cmake users to use the cmake provided by conda. Otherwise, we have to set the CMAKE_PREFIX_PATH properly.

I think in general you are using an "environment" so you need to let your operating system know about it.

It would be theoretically conceivable to use system cmake, but once you start jumping into conda-forge, why use anything else ;)

Regarding conda vs mamba, it shouldn't make a big difference these days, for me, it is just muscle memory. https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community

hmaarrfk commented 6 months ago

glad things are working!

hmaarrfk commented 6 months ago

I think that this might warrant a good page in our docs if it doesn't exist. i'm going to re-open to remember to do it at some time.

hmaarrfk commented 6 months ago

I can really understand how frustrating it must have been to be like "but i have cmake installed!!!"

vicentebolea commented 6 months ago

@hmaarrfk thanks!

Some users might be just like me that we are in env where we cannot really change the cmake that we use but depend in conda to install some libs. The workaround seems very simple:

prefix_path="$(conda info --json | jq '.active_prefix')"
export CMAKE_PREFIX_PATH="${prefix_path}:$CMAKE_PREFIX_PATH"
hmaarrfk commented 6 months ago

we cannot really change the cmake

can you elaborate on this scenerio? why must you use a custom cmake? (beyond developing features for CMake itself.)

vicentebolea commented 6 months ago

We provide our own cmake binaries for instance, or we depend in a particular version (we have a matrix of cmake versions that we want to try), or we use other package managers for the majority of the deps and conda for small subset of it.

hmaarrfk commented 6 months ago

Ah yes, this seems like a rather advanced workflow.

I would suggest maybe depending on conda-forge's cmake if you can. Try it out, it might be better than you think.

But regardless, if you want to write a little blurb about it that would be good.

For example, we have maintainer documentation (aimed at writing feedstocks) https://conda-forge.org/docs/maintainer/knowledge_base/#using-cmake

But maybe you want to write a section in the user documentation with your findings.

Integration within larger software is really important to me!