search

abseil / abseil-cpp

Abseil Common Libraries (C++)
https://abseil.io
Apache License 2.0
14.97k stars 2.62k forks source link

[Bug]: TESTONLY targets don't appear to get installed #1407

Open devjgm opened 1 year ago

devjgm commented 1 year ago

Describe the issue

I'm trying to build and install Abseil with CMake and use a PUBLIC target that's marked TESTONLY.

https://github.com/abseil/abseil-cpp/blob/c3b5022604551a045e383c68071d7be0a807839d/absl/log/CMakeLists.txt#L618-L620

After building and installing, I do not see a cmake target named absl::scoped_mock_log.

Steps to reproduce the problem

I fetch build and install abseil in a temporary location as follows:

$  git clone https://github.com/abseil/abseil-cpp
$ cd abseil-cpp
$ cmake -S. -Bcmake-out -DCMAKE_BUILD_TYPE=Release -GNinja -DCMAKE_CXX_STANDARD=14 -DABSL_BUILD_TESTING=ON -DABSL_USE_GOOGLETEST_HEAD=ON
$ cmake --build cmake-out
$ cmake --install cmake-out --prefix /tmp/absl

Then I look for a absl::scoped_mock_log CMake target

$ find /tmp/absl/ -name '*.cmake'
/tmp/absl//lib/cmake/absl/abslTargets.cmake
/tmp/absl//lib/cmake/absl/abslConfig.cmake
/tmp/absl//lib/cmake/absl/abslTargets-release.cmake
$ grep absl::scoped_mock /tmp/absl//lib/cmake/absl/abslTargets.cmake
$

You can grep that file for a target like, absl::optional to see that it does exist.

What version of Abseil are you using?

$ git show-ref HEAD d6ea4df62db59dc1ebc1fd3b62146700640ef400 refs/remotes/origin/HEAD

What operating system and version are you using?

macOS 13.2.1

What compiler and version are you using?

$ cc --version Apple clang version 14.0.0 (clang-1400.0.29.202) Target: arm64-apple-darwin22.3.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

What build system are you using?

$ cmake --version cmake version 3.25.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Additional context

From looking at the this file

https://github.com/abseil/abseil-cpp/blob/d6ea4df62db59dc1ebc1fd3b62146700640ef400/CMake/AbseilHelpers.cmake#L49-L50

I think I'm required to add ABSL_BUILD_TESTING=ON and BUILD_TESTING=ON (the default) in order to get TESTONLY targets to work.

I think this is probably a misfeature. I think every target that is considered a public API point that users may want to use should be built and installed by default, even without explicitly enabling testing.

derekmauro commented 1 year ago

Thanks for the report. I agree that the TESTONLY thing is a misfeature as well.

derekmauro commented 1 year ago

I took at look at this. It looks like the reason we require explicitly enabling testing to build and install the TESTONLY targets is because we don't need the GoogleTest dependency if we don't build the test helpers. There might be a better way of enabling this "feature" but for now I'm going to patch this to unblock you.

devjgm commented 1 year ago

Thanks.

I figured that avoiding a googletest dependency was probably the reasoning behind this decision. With a patch, we should be able to fix vcpkg, but it will require the vcpkg build Abseil's tests.

Would it be possible to make Abseil automatically build and install the TESTONLY targets when googletest is present? Because in order to use the TESTONLY targets, the user has to have googletest anyway. However, the user still probably doesn't care to build Abseil's tests.

derekmauro commented 1 year ago

The TESTONLY targets should now get installed when they are built.

We do have a CMake option ABSL_BUILD_TEST_HELPERS that should build the helpers but not the tests. I can look into turning that on automatically.

h-vetinari commented 1 year ago

@derekmauro, I used https://github.com/abseil/abseil-cpp/commit/807763a7f57dcf0ba4af7c3b218013e8f525e811 in https://github.com/conda-forge/abseil-cpp-feedstock/pull/58, and it seems that abseil_test_dll.dll is still not being installed even so. Is there another variable I'm overlooking to specify? I'm already setting

    -DBUILD_TESTING=ON
    -DABSL_BUILD_TEST_HELPERS=ON
    -DABSL_BUILD_TESTING=ON 
    -DABSL_USE_EXTERNAL_GOOGLETEST=ON
    -DABSL_FIND_GOOGLETEST=ON
h-vetinari commented 1 year ago

OK, nevermind, this was an operator error on my part (fun with default variable substitution not working in batch scripts... 😑). Still, it seem the test dll isn't buildable by default:

[228/445] Linking CXX executable bin\absl_random_beta_distribution_test.exe
FAILED: bin/absl_random_beta_distribution_test.exe 
cmd.exe /C "cd . && D:\bld\abseil-split_1682585378067\_build_env\Library\bin\cmake.exe -E vs_link_exe --intdir=absl\random\CMakeFiles\absl_random_beta_distribution_test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo absl\random\CMakeFiles\absl_random_beta_distribution_test.dir\beta_distribution_test.cc.obj  /out:bin\absl_random_beta_distribution_test.exe /implib:absl\random\absl_random_beta_distribution_test.lib /pdb:bin\absl_random_beta_distribution_test.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console  absl\abseil_test_dll.lib  D:\bld\abseil-split_1682585378067\_h_env\Library\lib\gmock.lib  D:\bld\abseil-split_1682585378067\_h_env\Library\lib\gtest_main.lib  absl\abseil_dll.lib  D:\bld\abseil-split_1682585378067\_h_env\Library\lib\gtest.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo absl\random\CMakeFiles\absl_random_beta_distribution_test.dir\beta_distribution_test.cc.obj /out:bin\absl_random_beta_distribution_test.exe /implib:absl\random\absl_random_beta_distribution_test.lib /pdb:bin\absl_random_beta_distribution_test.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console absl\abseil_test_dll.lib D:\bld\abseil-split_1682585378067\_h_env\Library\lib\gmock.lib D:\bld\abseil-split_1682585378067\_h_env\Library\lib\gtest_main.lib absl\abseil_dll.lib D:\bld\abseil-split_1682585378067\_h_env\Library\lib\gtest.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:bin\absl_random_beta_distribution_test.exe.manifest" failed (exit code 1120) with the following output:
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::ChiSquarePValue(double,int)" (?ChiSquarePValue@random_internal@lts_20230125@absl@@YANNH@Z) referenced in function "protected: bool __cdecl `anonymous namespace'::BetaDistributionTest::SingleChiSquaredTest<class absl::lts_20230125::beta_distribution<double> >(double,unsigned __int64,unsigned __int64)" (??$SingleChiSquaredTest@V?$beta_distribution@N@lts_20230125@absl@@@BetaDistributionTest@?A0x1270ec14@@IEAA_NN_K0@Z)
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "struct absl::lts_20230125::random_internal::DistributionMoments __cdecl absl::lts_20230125::random_internal::ComputeDistributionMoments(class absl::lts_20230125::Span<double const >)" (?ComputeDistributionMoments@random_internal@lts_20230125@absl@@YA?AUDistributionMoments@123@V?$Span@$$CBN@23@@Z) referenced in function "protected: bool __cdecl `anonymous namespace'::BetaDistributionTest::SingleZTestOnMeanAndVariance<class absl::lts_20230125::beta_distribution<double> >(double,unsigned __int64)" (??$SingleZTestOnMeanAndVariance@V?$beta_distribution@N@lts_20230125@absl@@@BetaDistributionTest@?A0x1270ec14@@IEAA_NN_K@Z)
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::ZScore(double,struct absl::lts_20230125::random_internal::DistributionMoments const &)" (?ZScore@random_internal@lts_20230125@absl@@YANNAEBUDistributionMoments@123@@Z) referenced in function "protected: bool __cdecl `anonymous namespace'::BetaDistributionTest::SingleZTestOnMeanAndVariance<class absl::lts_20230125::beta_distribution<double> >(double,unsigned __int64)" (??$SingleZTestOnMeanAndVariance@V?$beta_distribution@N@lts_20230125@absl@@@BetaDistributionTest@?A0x1270ec14@@IEAA_NN_K@Z)
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::RequiredSuccessProbability(double,int)" (?RequiredSuccessProbability@random_internal@lts_20230125@absl@@YANNH@Z) referenced in function "public: virtual void __cdecl `anonymous namespace'::BetaDistributionTest_TestSampleStatistics_Test::TestBody(void)" (?TestBody@BetaDistributionTest_TestSampleStatistics_Test@?A0x1270ec14@@UEAAXXZ)
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::MaxErrorTolerance(double)" (?MaxErrorTolerance@random_internal@lts_20230125@absl@@YANN@Z) referenced in function "protected: bool __cdecl `anonymous namespace'::BetaDistributionTest::SingleZTestOnMeanAndVariance<class absl::lts_20230125::beta_distribution<double> >(double,unsigned __int64)" (??$SingleZTestOnMeanAndVariance@V?$beta_distribution@N@lts_20230125@absl@@@BetaDistributionTest@?A0x1270ec14@@IEAA_NN_K@Z)
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "bool __cdecl absl::lts_20230125::random_internal::Near(class std::basic_string_view<char,struct std::char_traits<char> >,double,double,double)" (?Near@random_internal@lts_20230125@absl@@YA_NV?$basic_string_view@DU?$char_traits@D@std@@@std@@NNN@Z) referenced in function "protected: bool __cdecl `anonymous namespace'::BetaDistributionTest::SingleZTestOnMeanAndVariance<class absl::lts_20230125::beta_distribution<double> >(double,unsigned __int64)" (??$SingleZTestOnMeanAndVariance@V?$beta_distribution@N@lts_20230125@absl@@@BetaDistributionTest@?A0x1270ec14@@IEAA_NN_K@Z)
beta_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::BetaIncompleteInv(double,double,double)" (?BetaIncompleteInv@random_internal@lts_20230125@absl@@YANNNN@Z) referenced in function "protected: bool __cdecl `anonymous namespace'::BetaDistributionTest::SingleChiSquaredTest<class absl::lts_20230125::beta_distribution<double> >(double,unsigned __int64,unsigned __int64)" (??$SingleChiSquaredTest@V?$beta_distribution@N@lts_20230125@absl@@@BetaDistributionTest@?A0x1270ec14@@IEAA_NN_K0@Z)

bin\absl_random_beta_distribution_test.exe : fatal error LNK1120: 7 unresolved externals
h-vetinari commented 1 year ago

Still, it seem the test dll isn't buildable by default:

OK, I got a bit further with the following patch:

commit bfac9193f2b18fb585747df39c27ff94d3fce01d
Author: H. Vetinari <h.vetinari@gmx.com>
Date:   Sun May 7 18:24:46 2023 +1100

    add some necessary source files to ABSL_INTERNAL_TEST_DLL_FILES

diff --git a/CMake/AbseilDll.cmake b/CMake/AbseilDll.cmake
index 1aecd926..26078596 100644
--- a/CMake/AbseilDll.cmake
+++ b/CMake/AbseilDll.cmake
@@ -584,6 +584,10 @@ set(ABSL_INTERNAL_TEST_DLL_FILES
   "hash/hash_testing.h"
   "log/scoped_mock_log.cc"
   "log/scoped_mock_log.h"
+  "random/internal/chi_square.cc"
+  "random/internal/chi_square.h"
+  "random/internal/distribution_test_util.cc"
+  "random/internal/distribution_test_util.h"
   "random/internal/mock_helpers.h"
   "random/internal/mock_overload_set.h"
   "random/mocking_bit_gen.h"

which now fails with

[399/605] Linking CXX executable bin\absl_random_log_uniform_int_distribution_test.exe
FAILED: bin/absl_random_log_uniform_int_distribution_test.exe 
cmd.exe /C "cd . && D:\bld\abseil-split_1683452666531\_build_env\Library\bin\cmake.exe -E vs_link_exe --intdir=absl\random\CMakeFiles\absl_random_log_uniform_int_distribution_test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo absl\random\CMakeFiles\absl_random_log_uniform_int_distribution_test.dir\log_uniform_int_distribution_test.cc.obj  /out:bin\absl_random_log_uniform_int_distribution_test.exe /implib:absl\random\absl_random_log_uniform_int_distribution_test.lib /pdb:bin\absl_random_log_uniform_int_distribution_test.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console  D:\bld\abseil-split_1683452666531\_h_env\Library\lib\gmock.lib  D:\bld\abseil-split_1683452666531\_h_env\Library\lib\gtest_main.lib  absl\abseil_dll.lib  D:\bld\abseil-split_1683452666531\_h_env\Library\lib\gtest.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo absl\random\CMakeFiles\absl_random_log_uniform_int_distribution_test.dir\log_uniform_int_distribution_test.cc.obj /out:bin\absl_random_log_uniform_int_distribution_test.exe /implib:absl\random\absl_random_log_uniform_int_distribution_test.lib /pdb:bin\absl_random_log_uniform_int_distribution_test.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console D:\bld\abseil-split_1683452666531\_h_env\Library\lib\gmock.lib D:\bld\abseil-split_1683452666531\_h_env\Library\lib\gtest_main.lib absl\abseil_dll.lib D:\bld\abseil-split_1683452666531\_h_env\Library\lib\gtest.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:bin\absl_random_log_uniform_int_distribution_test.exe.manifest" failed (exit code 1120) with the following output:
log_uniform_int_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::ChiSquareValue(int,double)" (?ChiSquareValue@random_internal@lts_20230125@absl@@YANHN@Z) referenced in function "public: double __cdecl `anonymous namespace'::LogUniformIntChiSquaredTest::ChiSquaredTestImpl(void)" (?ChiSquaredTestImpl@LogUniformIntChiSquaredTest@?A0x0106261c@@QEAANXZ)
log_uniform_int_distribution_test.cc.obj : error LNK2019: unresolved external symbol "double __cdecl absl::lts_20230125::random_internal::ChiSquarePValue(double,int)" (?ChiSquarePValue@random_internal@lts_20230125@absl@@YANNH@Z) referenced in function "public: double __cdecl `anonymous namespace'::LogUniformIntChiSquaredTest::ChiSquaredTestImpl(void)" (?ChiSquaredTestImpl@LogUniformIntChiSquaredTest@?A0x0106261c@@QEAANXZ)

bin\absl_random_log_uniform_int_distribution_test.exe : fatal error LNK1120: 2 unresolved externals

AFAICT, the issue is that random_log_uniform_int_distribution_test depends on targets that are in both ABSL_INTERNAL_DLL_TARGETS & ABSL_INTERNAL_TEST_DLL_TARGETS, but only one of them gets passed to the link flags. Splitting up the line above, we see what gets added to the link flags:

LINK: command "[...]\link.exe /nologo [...] /subsystem:console
%LIBRARY_PATH%\gmock.lib
%LIBRARY_PATH%\gtest_main.lib
%LIBRARY_PATH%\gtest.lib
absl\abseil_dll.lib  # missing absl\abseil_test_dll.lib !
kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib
/MANIFEST [...]" failed (exit code 1120) with the following output...

I don't understand AbseilDll.cmake / AbseilHelpers.cmake well enough to tell why this is happening.

h-vetinari commented 1 year ago

I don't understand AbseilDll.cmake / AbseilHelpers.cmake well enough to tell why this is happening.

Found a fix after much banging of heads against walls:

commit dd5375ee269b6431c029efd1dae91525cb891492 (HEAD -> conda)
Author: H. Vetinari <h.vetinari@gmx.com>
Date:   Sun May 7 20:26:19 2023 +1100

    update ABSL_CC_TEST_LINKOPTS when using shared build

diff --git a/CMake/AbseilHelpers.cmake b/CMake/AbseilHelpers.cmake
index f452a676..89f6ba9a 100644
--- a/CMake/AbseilHelpers.cmake
+++ b/CMake/AbseilHelpers.cmake
@@ -427,6 +427,10 @@ function(absl_cc_test)
       DEPS ${ABSL_CC_TEST_DEPS}
       OUTPUT ABSL_CC_TEST_DEPS
     )
+    absl_internal_dll_targets(
+      DEPS ${ABSL_CC_TEST_LINKOPTS}
+      OUTPUT ABSL_CC_TEST_LINKOPTS
+    )
   else()
     target_compile_definitions(${_NAME}
       PUBLIC

That gets me until:

[526/605] Linking CXX executable bin\absl_cordz_handle_test.exe
FAILED: bin/absl_cordz_handle_test.exe 
cmd.exe /C "cd . && D:\bld\abseil-split_1683457518906\_build_env\Library\bin\cmake.exe -E vs_link_exe --intdir=absl\strings\CMakeFiles\absl_cordz_handle_test.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo absl\strings\CMakeFiles\absl_cordz_handle_test.dir\internal\cordz_handle_test.cc.obj  /out:bin\absl_cordz_handle_test.exe /implib:absl\strings\absl_cordz_handle_test.lib /pdb:bin\absl_cordz_handle_test.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console  absl\abseil_test_dll.lib  D:\bld\abseil-split_1683457518906\_h_env\Library\lib\gmock_main.lib  absl\abseil_dll.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo absl\strings\CMakeFiles\absl_cordz_handle_test.dir\internal\cordz_handle_test.cc.obj /out:bin\absl_cordz_handle_test.exe /implib:absl\strings\absl_cordz_handle_test.lib /pdb:bin\absl_cordz_handle_test.pdb /version:0.0 /machine:x64 /INCREMENTAL:NO /subsystem:console absl\abseil_test_dll.lib D:\bld\abseil-split_1683457518906\_h_env\Library\lib\gmock_main.lib absl\abseil_dll.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:bin\absl_cordz_handle_test.exe.manifest" failed (exit code 1120) with the following output:
cordz_handle_test.cc.obj : error LNK2019: unresolved external symbol "const absl::lts_20230125::cord_internal::CordzHandle::`vftable'" (??_7CordzHandle@cord_internal@lts_20230125@absl@@6B@) referenced in function "public: __cdecl absl::lts_20230125::cord_internal::CordzHandle::CordzHandle(void)" (??0CordzHandle@cord_internal@lts_20230125@absl@@QEAA@XZ)
bin\absl_cordz_handle_test.exe : fatal error LNK1120: 1 unresolved externals

Missing vtables is much more annoying to debug because it could be any member function or data member. I tried to have a look and no missing definition in the header/implementation of that class stood out to me. It seems to point to the constructor, but if I understand member initializer lists correctly, CordzHandle() : CordzHandle(false) {} is a definition and not just a declaration.

The one thing I'm not sure about is how the constructor definition

ABSL_CONST_INIT CordzHandle::Queue CordzHandle::global_queue_(absl::kConstInit);

seems to simultaneously try to address two declarations (constexpr explicit Queue(absl::ConstInitType) : mutex(...) {} & ABSL_CONST_INIT static Queue global_queue_;), my thinking is that Queue(absl::ConstInitType) : mutex(...) {} might not be a "full" constructor definition for Queue since it only defines the concurrent definition of a class member, but not all. Not sure if that's the reason, but I'm very far out of my depths here, this is way above my C++ foo.

h-vetinari commented 1 year ago

Missing vtables is much more annoying to debug because it could be any member function or data member.

OK, I had managed to lead myself astray here, it was one of those ABSL_DLL cases after all. I opened a PR with the fixes that were necessary to do shared builds including test targets on windows: https://github.com/abseil/abseil-cpp/pull/1445