LLVM utilities

[owen-oqc]

Comment:

Hi I'm in the process of swapping out a from-source LLVM build with the llvmdev packages.

utilities (count, not, FileCheck etc) end up in a non default location here and that's caused some problems for us particularly with the cmake function add_lit_testsuite where FileCheck for example is a dependency that cannot otherwise be found. What's the rational with the current design?

[h-vetinari]

utilities (count, not, FileCheck etc) end up in a non default location here

Your link (currently) points to -DLLVM_INCLUDE_DOCS=OFF, is that what you meant? I doubt that utilities would be toggled by a switch called docs... 🤔

What's the rationale with the current design?

Basically: we don't want consumers of packages compiled with llvmdev to have to pull in the whole shebang. That's why the library has a separate output, that's why llvm-tools are separate (add them if you need them), plus lit (because it pulls in python, we don't want everyone to pull in the whole python stack, so it's separate), plus some extra bits to ensure we are hooking up things like run-export correctly.

Beyond that, there's no fundamental reason why normal usage of llvm shouldn't work in our setup, and if we can do it without breaking our other constraints, we should! In particular, despite the separation into separate outputs, llvmdev should be unifying everything back together as if it had all been installed in one piece.

Can you be more specific about which pieces are in the wrong place for you? Also, can you please specify the output of your conda list, as the issue template asks?

[owen-oqc]

Apologies for the incorrect link line, that should be pointing to line 53 for the utils install directory

Looking at the llvm-project I believe if unspecified these tools would end up in CMAKE_INSTALL_BINDIR. To be clear i'm not saying that this is wrong, but there might be assumptions in some of the LLVM internal cmake macros/functions about where these tools are placed as uncovered with the add_lit_testsuite function

#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
c-ares                    1.19.1               hb547adb_0    conda-forge
ca-certificates           2023.7.22            hf0a4a13_0    conda-forge
cmake                     3.26.4               hc0af03a_0    conda-forge
expat                     2.5.0                hb7217d7_1    conda-forge
icu                       72.1                 he12128b_0    conda-forge
krb5                      1.21.1               h92f50d5_0    conda-forge
libcurl                   8.2.1                hc52a3a8_0    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libiconv                  1.17                 he4db4b2_0    conda-forge
libllvm16                 16.0.6               he79909e_1    conda-forge
libmlir                   16.0.6               h2d3518b_0    conda-forge
libmlir16                 16.0.6               h2d3518b_0    conda-forge
libnghttp2                1.52.0               hae82a92_0    conda-forge
libsqlite                 3.42.0               hb31c410_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libuv                     1.44.2               hb547adb_1    conda-forge
libxml2                   2.11.4               he3bdae6_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
lit                       16.0.6             pyh1a96a4e_1    conda-forge
llvm-tools                16.0.6               he79909e_1    conda-forge
llvmdev                   16.0.6               h504e6bf_1    conda-forge
mlir                      16.0.6               h2d3518b_0    conda-forge
ncurses                   6.4                  h7ea286d_0    conda-forge
ninja                     1.11.1               hffc8910_0    conda-forge
openssl                   3.1.2                h53f4e23_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.9.16          hea58f1e_0_cpython    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
rhash                     1.4.3                hb547adb_1    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
zlib                      1.2.13               h53f4e23_5    conda-forge
zstd                      1.5.2                h4f39d0f_7    conda-forge

[h-vetinari]

OK, looking at LLVM_UTILS_INSTALL_DIR=libexec/llvm, this an aspect of the setup here that I do not know. It was added by @isuruf in https://github.com/conda-forge/llvmdev-feedstock/commit/43b6479d0d7bbe8ef30979b1a8b3ab337307c187 in the context of #74, which also got rid of the llvm-utils output.

There not much information to go by in that PR, perhaps Isuru can explain? Context here is that some utilities are broken with our setup here.

[isuruf]

That directory is chosen so that they don't interfere with the system binaries like not. They have too generic names. Downstream projects should not assume that the two directories LLVM_UTILS_INSTALL_DIR and LLVM_INSTALL_BINDIR are equal.

[h-vetinari]

Thanks for the quick response! I'm not proposing to change the current setup, just trying to understand if it's possible to fix stuff like add_lit_testsuite (see OP) with our setup.

It seems that LLVM itself is not ready to have LLVM_UTILS_INSTALL_DIR != LLVM_INSTALL_BINDIR? Guess there might be some inaccuracies in the upstream CMake files...

[owen-oqc]

I agree, upstream in LLVM where the fixes need to go. Thanks for discussing - I don't think there are any changes required on this project for this.

[h-vetinari]

BTW, it's possible to formulate a test here (some call to add_list_testsuite for example), develop a patch that fixes the test, and then (optionally) try to upstream the patch.