search

protocolbuffers / protobuf

Protocol Buffers - Google's data interchange format
http://protobuf.dev
Other
65.59k stars 15.48k forks source link

23.0: issue with generated protobuf.pc files (it depends on `utf8_range`) #12746

Closed kloczek closed 10 months ago

kloczek commented 1 year ago

Looks like generated protobuf.pc has in Requires utf8_range.

prefix=/usr
exec_prefix=/usr
libdir=/usr/lib64
includedir=/usr/include

Name: Protocol Buffers
Description: Google's Data Interchange Format
Version: 23.0.0
Requires: absl_absl_check absl_absl_log absl_algorithm absl_base absl_bind_front absl_bits absl_btree absl_cleanup absl_cord absl_core_headers absl_debugging absl_die_if_null absl_dynamic_annotations absl_flags absl_flat_hash_map absl_flat_hash_set absl_function_ref absl_hash absl_layout absl_log_initialize absl_log_severity absl_memory absl_node_hash_map absl_node_hash_set absl_optional absl_span absl_status absl_statusor absl_strings absl_synchronization absl_time absl_type_traits absl_utility absl_variant utf8_range
Libs: -L${libdir} -lprotobuf
Cflags: -I${includedir} -DPROTOBUF_USE_DLLS
Conflicts: protobuf-lite
kloczek commented 1 year ago

In generated protobuf-config.cmake I see as well:

# User options
include("${CMAKE_CURRENT_LIST_DIR}/protobuf-options.cmake")

# Depend packages

if(NOT TARGET absl::strings)
  find_package(absl CONFIG)
endif()
if(NOT TARGET utf8_range)
  find_package(utf8_range CONFIG)
endif()

# Imported targets
include("${CMAKE_CURRENT_LIST_DIR}/protobuf-targets.cmake")

# protobuf-generate function
include("${CMAKE_CURRENT_LIST_DIR}/protobuf-generate.cmake")

# CMake FindProtobuf module compatible file
if(protobuf_MODULE_COMPATIBLE)
  include("${CMAKE_CURRENT_LIST_DIR}/protobuf-module.cmake")
endif()
kloczek commented 1 year ago

Just in case I've build protobuf with -D utf8_range_ENABLE_INSTALL=OFF.

kloczek commented 1 year ago

I've added quick hack to remove utf8_range from Requires on rpm package %install stage. With such product I was able to start testing use generated package on building other packages. Results:

[..] 



In other words looks likke protobuf 23.0 is not ready to be used.
kloczek commented 1 year ago

Gentle ping .. any update?

I was able to rebuild protobuf-c usimg 22.5 however still on rebuilding libphonenumber, mysql, opencv are failing on missing symbols.

fowles commented 1 year ago

@coryan do you have a moment to take a look at this?

coryan commented 1 year ago

Happy to do a drive-by. It would be useful to know what platform (Linux, Windows, etc.) this was on, and what version of the compiler. Looks like some Linux distro, with GCC.

The dependency on utf8_range

Is utf8_range a hard dependency? If it is, then the install instructions should say so. Consider adding step-by-step instructions to build all the dependencies from source in https://github.com/protocolbuffers/protobuf/blob/main/src/README.md

If it is not a hard dependency, then this:

https://github.com/protocolbuffers/protobuf/blob/56d1b0f3f5e0170ea336a754af0bb3ae7b6d4bfa/cmake/install.cmake#L7

Needs to have a conditional:

if (some_variable_enabled_when_utf8_is_required)
    list(APPEND _pc_targets "utf8_range")
endif ()

The broken build for libphonenumber

This error:

/usr/bin/ld: /tmp/ccOihmdo.lto.o: in function `google::protobuf::RepeatedField<int>::GrowNoAnnotate(int, int)':
/usr/include/google/protobuf/thread_safe_arena.h:198: undefined reference to `google::protobuf::internal::ThreadSafeArena::thread_cache_'

was fixed in #12700. I think @fowles already backported that fix to the 22.x and 23.x branches.

The broken build for mysql

This error

/usr/bin/ld: /usr/lib64/libprotoc.so: undefined reference to symbol '_ZN4absl12lts_2023012513base_internal12SpinLockWaitEPSt6atomicIjEiPKNS1_22SpinLockWaitTransitionENS1_14SchedulingModeE'
/usr/bin/ld: /usr/lib64/libabsl_spinlock_wait.so.2301.0.0: error adding symbols: DSO missing from command line

Seems like a missing -labsl*. Most likely mysql is using find_package(protobuf), using the FindProtobuf embedded with CMake and missing all the new deps. That is indeed the case:

https://github.com/mysql/mysql-server/blob/ea7087d885006918ad54458e7aad215b1650312c/cmake/protobuf.cmake#L124

That needs to change to find_package(protobuf CONFIG). See #12292. It is becoming a FAQ, the Protobuf team should consider updating the release notes.

The broken build for opencv

I think in this case the error explains itself:

   79 | #error "C++ versions less than C++14 are not supported."

The opencv library was configured (or maybe defaults to) compile as C++11:

[ 21%] Building CXX object modules/dnn/CMakeFiles/opencv_dnn.dir/opencv-caffe.pb.cc.o
cd /home/tkloczko/rpmbuild/BUILD/opencv-4.7.0/x86_64-redhat-linux-gnu/modules/dnn && /usr/bin/g++ \
... stuff omitted for clarity ...
 -DNDEBUG \
 -std=c++11 \
 -fPIC -Wno-suggest-override -Wno-array-bounds -MD -MT modules/dnn/CMakeFiles/opencv_dnn.dir/opencv-caffe.pb.cc.o -MF CMakeFiles/opencv_dnn.dir/opencv-caffe.pb.cc.o.d -o CMakeFiles/opencv_dnn.dir/opencv-caffe.pb.cc.o -c /home/tkloczko/rpmbuild/BUILD/opencv-4.7.0/x86_64-redhat-linux-gnu/modules/dnn/opencv-caffe.pb.cc

I am not familiar with the configuration options for opencv. It may be necessary to use cmake -DCMAKE_CXX_STANDARD=14 ..., or maybe just omit the -DCMAKE_CXX_STANDARD=14 altogether. Most compilers default to C++14 or higher these days. There are very few cases where one needs to explicitly request C++11.

In summary:

HTH

kloczek commented 1 year ago

Happy to do a drive-by. It would be useful to know what platform (Linux, Windows, etc.) this was on, and what version of the compiler. Looks like some Linux distro, with GCC.

The dependency on utf8_range

Is utf8_range a hard dependency? If it is, then the install instructions should say so. Consider adding step-by-step instructions to build all the dependencies from source in https://github.com/protocolbuffers/protobuf/blob/main/src/README.md

If it is not a hard dependency, then this:

https://github.com/protocolbuffers/protobuf/blob/56d1b0f3f5e0170ea336a754af0bb3ae7b6d4bfa/cmake/install.cmake#L7

Needs to have a conditional:

if (some_variable_enabled_when_utf8_is_required)
    list(APPEND _pc_targets "utf8_range")
endif ()

I dont see any fix for that in 22.x branch https://github.com/protocolbuffers/protobuf/compare/v22.5...22.x

The broken build for libphonenumber

This error:

/usr/bin/ld: /tmp/ccOihmdo.lto.o: in function `google::protobuf::RepeatedField<int>::GrowNoAnnotate(int, int)':
/usr/include/google/protobuf/thread_safe_arena.h:198: undefined reference to `google::protobuf::internal::ThreadSafeArena::thread_cache_'

was fixed in #12700. I think @fowles already backported that fix to the 22.x and 23.x branches.

None of the commits from https://github.com/protocolbuffers/protobuf/pull/12700 are on the list https://github.com/protocolbuffers/protobuf/compare/v22.5...22.x

The broken build for mysql

This error

/usr/bin/ld: /usr/lib64/libprotoc.so: undefined reference to symbol '_ZN4absl12lts_2023012513base_internal12SpinLockWaitEPSt6atomicIjEiPKNS1_22SpinLockWaitTransitionENS1_14SchedulingModeE'
/usr/bin/ld: /usr/lib64/libabsl_spinlock_wait.so.2301.0.0: error adding symbols: DSO missing from command line

Seems like a missing -labsl*. Most likely mysql is using find_package(protobuf), using the FindProtobuf embedded with CMake and missing all the new deps. That is indeed the case:

https://github.com/mysql/mysql-server/blob/ea7087d885006918ad54458e7aad215b1650312c/cmake/protobuf.cmake#L124

That needs to change to find_package(protobuf CONFIG). See #12292. It is becoming a FAQ, the Protobuf team should consider updating the release notes.

mysql is not using absl API and error message shows missing symbol in libprotoc -> it points on libprotoc underlinking with some of the absl DSOs.

[tkloczko@pers-jacek SPECS]$ objdump -x /usr/lib64/libprotoc.so | grep NEED
  NEEDED               libprotobuf.so.22.5.0
  NEEDED               libabsl_log_internal_check_op.so.2301.0.0
  NEEDED               libabsl_log_internal_message.so.2301.0.0
  NEEDED               libabsl_log_internal_nullguard.so.2301.0.0
  NEEDED               libabsl_hash.so.2301.0.0
  NEEDED               libabsl_raw_hash_set.so.2301.0.0
  NEEDED               libabsl_str_format_internal.so.2301.0.0
  NEEDED               libabsl_synchronization.so.2301.0.0
  NEEDED               libabsl_strings.so.2301.0.0
  NEEDED               libabsl_throw_delegate.so.2301.0.0
  NEEDED               libabsl_spinlock_wait.so.2301.0.0
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  VERNEED              0x0000000000014270
  VERNEEDNUM           0x0000000000000003

In summary:

  • Looks like the version of Protobuf you are using is missing some fixes already in the release branches.

As I've wrote I've tested that with 23.0.

  • Maybe there is a bug in the CMake files if utf8_range is not a hard-dependency.
  • I think there are multiple opportunities to improve the Protobuf documentation around building and installation.
  • mysql needs some patches to use Protobuf >= 22.0.

I disagree. It compiles and linking fails because missing symbols.

coryan commented 1 year ago

Happy to do a drive-by. It would be useful to know what platform (Linux, Windows, etc.) this was on, and what version of the compiler. Looks like some Linux distro, with GCC.

The dependency on utf8_range

Is utf8_range a hard dependency? If it is, then the install instructions should say so. Consider adding step-by-step instructions to build all the dependencies from source in https://github.com/protocolbuffers/protobuf/blob/main/src/README.md If it is not a hard dependency, then this: https://github.com/protocolbuffers/protobuf/blob/56d1b0f3f5e0170ea336a754af0bb3ae7b6d4bfa/cmake/install.cmake#L7

I dont see any fix for that in 22.x branch v22.5...22.x

Yes. I said "if it is not a hard dependency". But I looked at the code again and it is. That means this:

I've added quick hack to remove utf8_range from Requires on rpm package %install stage.

Was probably not a good idea.

was fixed in #12700. I think @fowles already backported that fix to the 22.x and 23.x branches.

None of the commits from #12700 are on the list v22.5...22.x

I am not sure why that matters since you are testing with 23.0. FWIW, the relevant commit in the 22.x series is https://github.com/protocolbuffers/protobuf/commit/18fae1c15112efad2080c2b2f726d904fea48b35 and it is there:

https://github.com/protocolbuffers/protobuf/compare/v22.4...22.x

The relevant commit in the 23.x series is https://github.com/protocolbuffers/protobuf/commit/1ca4e9c4859a23112684138c78608ddc0b8f1770

mysql is not using absl API and error message shows missing symbol in libprotoc -> it points on libprotoc underlinking with some of the absl DSOs.

I am not sure I follow. That is intentional. libprotoc.so depends on the Abseil symbols. Some of the abseil symbols are part of the public API of libprotoc. It is not expected that -lprotoc by itself will be enough. If it did (for example by linking all the absl* libraries into the .so), I think applications that used Abseil and libprotoc at the same time would quickly run into ODR violations.

In summary:

  • Looks like the version of Protobuf you are using is missing some fixes already in the release branches.

As I've wrote I've tested that with 23.0.

Thanks that is useful. Could you also share the specific Linux distro and the steps you are using to compile Protobuf? It would make it easier to repro any bugs locally.

  • mysql needs some patches to use Protobuf >= 22.0.

I disagree. It compiles and linking fails because missing symbols.

I think you are expecting that linking with just -lprotoc should work. It does not and it is not expected to. The Protobuf team has fixed the pkg-config modules and the CMake config files to add any new -l flags. That should make it easier to migrate build scripts to add the flags where needed.

Build scripts written for Protobuf <= 21.x must be updated to take into account the new dependencies. If a build script (like mysql's) relies on CMake's FindProtobuf module it needs to be updated. That is because FindProtobuf does not know about the new dependencies.

mysql uses that module and that is why the build is missing -l flags.

I also do not think the Protobuf team should (or could) fix the FindProtobuf module. First, because they cannot go back in time and fix the version of that module that already shipped everywhere. Second, because CMake's team plan of record is to migrate to the config files that Protobuf already generates and installs.

I appreciate this is maybe surprising and/or frustrating. But I am not sure what actions are you suggesting.

kloczek commented 1 year ago

I am not sure why that matters since you are testing with 23.0. FWIW, the relevant commit in the 22.x series is https://github.com/protocolbuffers/protobuf/commit/18fae1c15112efad2080c2b2f726d904fea48b35 and it is there:

Nope I'm now on 22.5 because with that version I was able to compile everythiong and have now only linking issues.

kloczek commented 1 year ago

Trying to figure out why libprotobuf DSO does not provdes `google::protobuf::internal::ThreadSafeArena::threadcache' I found another "intersting: thing. Despite

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ cmake -L
CMake Warning:
  No source or binary directory provided.  Both will be assumed to be the
  same as the current working directory, but note that this warning will
  become a fatal error in future CMake releases.

CMake Error: The source directory "/home/tkloczko/rpmbuild/BUILD/protobuf-22.5/x86_64-redhat-linux-gnu" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
-- Cache values
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr
GTest_DIR:PATH=/usr/lib64/cmake/GTest
absl_DIR:PATH=/usr/lib64/cmake/absl
protobuf_ABSL_PROVIDER:STRING=package
protobuf_ALLOW_CCACHE:BOOL=OFF
protobuf_BUILD_CONFORMANCE:BOOL=OFF
protobuf_BUILD_EXAMPLES:BOOL=OFF
protobuf_BUILD_LIBPROTOC:BOOL=OFF
protobuf_BUILD_PROTOBUF_BINARIES:BOOL=ON
protobuf_BUILD_PROTOC_BINARIES:BOOL=ON
protobuf_BUILD_SHARED_LIBS:BOOL=ON
protobuf_BUILD_TESTS:BOOL=ON
protobuf_DISABLE_RTTI:BOOL=OFF
protobuf_INSTALL:BOOL=ON
protobuf_INSTALL_EXAMPLES:BOOL=OFF
protobuf_REMOVE_INSTALLED_HEADERS:BOOL=OFF
protobuf_TEST_XML_OUTDIR:BOOL=OFF
protobuf_USE_EXTERNAL_GTEST:BOOL=ON
protobuf_WITH_ZLIB:BOOL=ON
utf8_range_ENABLE_INSTALL:BOOL=OFF
utf8_range_ENABLE_TESTS:BOOL=OFF

in which is possible to see protobuf_BUILD_LIBPROTOC:BOOL=OFF libprotoc DSO is build, linked and installed.

kloczek commented 1 year ago

I am not sure why that matters since you are testing with 23.0. FWIW, the relevant commit in the 22.x series is 18fae1c and it is there:

Nope I'm now on 22.5 because with that version I was able to compile everythiong and have now only linking issues.

BTW looks like 18fae1c1 is already included in 22.5.

kloczek commented 1 year ago

According to https://github.com/protocolbuffers/protobuf/blob/17a2d53176bc5c3bc3be8c636f999ca7a645cc68/cmake/README.md?plain=1#L296-L314 PROTOBUF_USE_DLLS define should be used only on Win. Going to try to roll back 18fae1c1.

coryan commented 1 year ago

PROTOBUF_USE_DLLS define should be used only on Win.

I think the documentation is very much out of date and I filed a bug requesting fixes (see #12787).

Going to try to roll back 18fae1c.

If you compile the library with PROTOBUF_USE_DLLS defined and use it with that undefined you will have an inconsistent definition for this class:

https://github.com/protocolbuffers/protobuf/blob/17a2d53176bc5c3bc3be8c636f999ca7a645cc68/src/google/protobuf/thread_safe_arena.h#L264-L271

And that macro is defined when CMake creates shared libraries. Both on Windows and other platforms:

https://github.com/protocolbuffers/protobuf/blob/42b20b3a64d1e327c9f8e67782b3c45b6f5b59c1/cmake/libprotobuf.cmake#L30-L34

That is what causes the undefined google::protobuf::internal::ThreadSafeArena::thread_cache_ symbol.

Are you sure you are using the correct .pc file when building the downstream dependency? Or if you are not using .pc files and you are using CMake: is the downstream build using the protobuf config (as opposed to the FindProtobuf module)?

kloczek commented 1 year ago

Are you sure you are using the correct .pc file when building the downstream dependency? Or if you are not using .pc files and you are using CMake: is the downstream build using the protobuf config (as opposed to the FindProtobuf module)?

Yes I'm sure. Generated .pc files have that -DPROTOBUF_USE_DLLS (I've done that without checking possible impact of that kind of change). Of course reverte 18fae1c caused that it was not possible to build protobuf baceuse other sybols in that case where missing. Nevertheless looks liks like something is wrong around use that define in some headers.

kloczek commented 1 year ago

Or if you are not using .pc files and you are using CMake: is the downstream build using the protobuf config (as opposed to the FindProtobuf module)?

opencv, libphonenumber and Oracle mysql 8.x are usimg pkgconfig so what is in cmake module is not relevant in those cases.

kloczek commented 1 year ago

Gentle ping .. any update? 🤔

mkruskal-google commented 1 year ago

I'm having trouble figuring out where this issue is at. Did you add back the dependency on utf8_range? That is a required (private) dependency, and you'll definitely break the build by stripping it out

kloczek commented 1 year ago

Did you add back the dependency on utf8_range?

Yes. On packaging after installation in DESTDIR I'm usimg sed to remove " utf8_range" from .pc files.

That is a required (private) dependency, and you'll definitely break the build by stripping it out

Af far as I've been looking utf8_range it used out of third_party/utf8_range/ and as I wrote I'm building protobuf with -D utf8_range_ENABLE_INSTALL=OFF and despite that utf8_range is added in .pc files Requires:

mkruskal-google commented 1 year ago

Yea that's the problem. We require utf8_range, and if you set utf8_range_ENABLE_INSTALL=OFF we won't have it available. You need to install it and keep the dependency in the .pc files. We may collapse upb and utf8_range into the protobuf repo in a future release, but for 22.x and 23.x you need both

kloczek commented 1 year ago

Yea that's the problem. We require utf8_range, and if you set utf8_range_ENABLE_INSTALL=OFF we won't have it available.

It does't matter. Whatever is used as bundled library should not be listed in .pc files.

mkruskal-google commented 1 year ago

it's not used as a bundled library today. It needs to be installed and linked for protobuf to work.

At least the mysql issue is caused by mysql using find_package(protobuf). CMake's FindProtobuf is broken for 22.x and up, and find_package(protobuf CONFIG) needs to be used. Related to that, mysql also seems to be using all of our vendored logging code that's been replaced by Abseil.

kloczek commented 1 year ago

I have no installed utf8_range and just tested 23.1 it still has in .pc files listed utf8_range in Requires.

mkruskal-google commented 1 year ago

Yes, because we do require utf8_range. We ship it as a git sub-tree, and install it during protobuf installation. If you explicitly disable this installation you'll break our library

kloczek commented 1 year ago

Yes, because we do require utf8_range. We ship it as a git sub-tree, and install it during protobuf installation. If you explicitly disable this installation you'll break our library

Just started plaing with 23.1 and:

Did you try to build latest Oracle mysql, opencv and libpronenumber with installes 23.1? In all three cases build fails on that missing symbol.

And again: as I'm building protobuf with bundled utf8_range why this library appears on pkgconfig dependencies list if none of the protobuf installed header files are usong anything from utf8_range?

kloczek commented 1 year ago

Other thing is question about using absl. mysql and libpronenumber are using boost (but only header files without libking with any boost DSOs) and libstdc++ classes. opencv is using only libstdc++. Additionally absl has very fragmented ABI which consist out of MANY DSOs which makes LTO inefficient and adds more wasted memory on jump tables between DSOs symbols.

Is it really necessary tu use it? 🤔

fowles commented 1 year ago

Yes, absl is definitely required for protobuf. We use it for a variety of important internal optimizations and in 23.x we add support for bytes fields with type absl::Cord, which is very important for zero copy handling of large blobs.

kloczek commented 1 year ago

Yes, absl is definitely required for protobuf. We use it for a variety of important internal optimizations and in 23.x we add support for bytes fields with type absl::Cord, which is very important for zero copy handling of large blobs.

If you say so ..

So what about taht missing symbol?

[tkloczko@pers-jacek protobuf-23.1]$ grep -r thread_cache_
src/google/protobuf/arena.cc:  static internal::ThreadLocalStorage<ThreadCache>* thread_cache_ =
src/google/protobuf/arena.cc:  return *thread_cache_->Get();
src/google/protobuf/arena.cc:    ThreadSafeArena::ThreadCache ThreadSafeArena::thread_cache_;
src/google/protobuf/thread_safe_arena.h:  PROTOBUF_CONSTINIT static PROTOBUF_THREAD_LOCAL ThreadCache thread_cache_;
src/google/protobuf/thread_safe_arena.h:  static ThreadCache& thread_cache() { return thread_cache_; }

As you see that class is not part of the utf8_range.

mkruskal-google commented 1 year ago

I don't think the original problem has anything to do with utf8_range, it was just some of the subsequent ones you saw when you tried to remove the dependency. I think Carlos nailed the root problem a while back:

PROTOBUF_USE_DLLS define should be used only on Win.

I think the documentation is very much out of date and I filed a bug requesting fixes (see #12787).

Going to try to roll back 18fae1c.

If you compile the library with PROTOBUF_USE_DLLS defined and use it with that undefined you will have an inconsistent definition for this class:

https://github.com/protocolbuffers/protobuf/blob/17a2d53176bc5c3bc3be8c636f999ca7a645cc68/src/google/protobuf/thread_safe_arena.h#L264-L271

And that macro is defined when CMake creates shared libraries. Both on Windows and other platforms:

https://github.com/protocolbuffers/protobuf/blob/42b20b3a64d1e327c9f8e67782b3c45b6f5b59c1/cmake/libprotobuf.cmake#L30-L34

That is what causes the undefined google::protobuf::internal::ThreadSafeArena::thread_cache_ symbol.

Are you sure you are using the correct .pc file when building the downstream dependency? Or if you are not using .pc files and you are using CMake: is the downstream build using the protobuf config (as opposed to the FindProtobuf module)?

He confirmed that at least mysql is using FindProtobuf, meaning that they'd have an inconsistent value of PROTOBUF_USE_DLLS and it would cause issues with this symbol

fowles commented 1 year ago

That is usually caused by some configuration error with support for thread local

https://github.com/protocolbuffers/protobuf/blob/284a057ac199c28620ad9fa95508861f349588c4/src/google/protobuf/port_def.inc#LL1079C8-L1079C31

https://github.com/protocolbuffers/protobuf/blob/284a057ac199c28620ad9fa95508861f349588c4/src/google/protobuf/port_def.inc#L658-L665

are likely related. What platform are you on?

kloczek commented 1 year ago

If you compile the library with PROTOBUF_USE_DLLS defined and use it with that undefined you will have an inconsistent definition for this class:

Here cmake parames which I have in my protobuf.spec

%build
%cmake \
        -D CMAKE_PREFIX_PATH=%{_prefix} \
        -D protobuf_ABSL_PROVIDER=ON \
        -D protobuf_ABSL_PROVIDER=package \
        -D protobuf_BUILD_TESTS=ON \
        -D protobuf_USE_EXTERNAL_GTEST=ON \
        -D utf8_range_ENABLE_INSTALL=OFF \
        %{nil}

protobuf copiles and links with statis libutf8_validity.a which is not installed.

kloczek commented 1 year ago

He confirmed that at least mysql is using FindProtobuf, meaning that they'd have an inconsistent value of PROTOBUF_USE_DLLS and it would cause issues with this symbol

Negative ..

[tkloczko@pers-jacek mysql-8.0.33]$ grep -r FindProtobuf
cmake/protobuf.cmake:  # Set the same variables as FindProtobuf.cmake (First reset all)
extra/protobuf/protobuf-3.19.4/cmake/protobuf-config.cmake.in:# CMake FindProtobuf module compatible file
extra/protobuf/protobuf-3.19.4/cmake/protobuf-options.cmake:# FindProtobuf module compatible
extra/protobuf/protobuf-3.19.4/cmake/protobuf-options.cmake:option(protobuf_MODULE_COMPATIBLE "CMake built-in FindProtobuf.cmake module compatible" OFF)

In libphonenumber

[tkloczko@pers-jacek libphonenumber-8.13.11]$ grep -r FindProtobuf
[tkloczko@pers-jacek libphonenumber-8.13.11]$

and opencv

[tkloczko@pers-jacek opencv-4.7.0]$ grep -r FindProtobuf
CMakeLists.txt:include(cmake/OpenCVFindProtobuf.cmake)
kloczek commented 1 year ago

That is usually caused by some configuration error with support for thread local

protobuf cmake output:

+ cd protobuf-23.1
+ ASMFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ CFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ CXXFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none'
+ FFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules'
+ FCFLAGS='-O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -I/usr/lib64/gfortran/modules'
+ LDFLAGS='-Wl,--gc-sections -Wl,--as-needed -flto=auto -flto-partition=none -fuse-linker-plugin -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,--build-id=sha1'
+ RUSTFLAGS='-C codegen-units=1 -C debuginfo=2 -C opt-level=2 -C link-arg=-fdata-sections -C link-arg=-ffunction-sections -C link-arg=-Wl,--as-needed -C link-arg=-Wl,-z,now -C link-arg=-Wl,-z,relro --cap-lints=warn'
+ VALAFLAGS=-g
+ CC=/usr/bin/gcc
+ CXX=/usr/bin/g++
+ FC=/usr/bin/gfortran
+ AR=/usr/bin/gcc-ar
+ NM=/usr/bin/gcc-nm
+ RANLIB=/usr/bin/gcc-ranlib
+ export ASMFLAGS CFLAGS CXXFLAGS FFLAGS FCFLAGS LDFLAGS VALAFLAGS CC CXX FC AR NM RANLIB RUSTFLAGS VALAFLAGS
+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S . -D CMAKE_PREFIX_PATH=/usr -D protobuf_ABSL_PROVIDER=ON -D protobuf_ABSL_PROVIDER=package -D protobuf_BUILD_TESTS=ON -D protobuf_USE_EXTERNAL_GTEST=ON -D utf8_range_ENABLE_INSTALL=OFF
-- The C compiler identification is GNU 13.1.1
-- The CXX compiler identification is GNU 13.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
--
-- 23.1.0
-- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT
-- Performing Test protobuf_HAVE_LD_VERSION_SCRIPT - Success
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS
-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success
-- Configuring done (1.7s)
-- Generating done (0.1s)
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_Fortran_FLAGS_RELEASE
    INCLUDE_INSTALL_DIR
    LIB_INSTALL_DIR
    LIB_SUFFIX
    SHARE_INSTALL_PREFIX
    SYSCONF_INSTALL_DIR

-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/protobuf-23.1/x86_64-redhat-linux-gnu
mkruskal-google commented 1 year ago

That grep won't turn up FindPackage uses. The key point is that FIND_PACKAGE(Protobuf) falls back to the FindPackage implementation shipped with cmake, which is broken. What you want is FIND_PACKAGE(Protobuf CONFIG)

kloczek commented 1 year ago 
[tkloczko@pers-jacek BUILD]$ grep -ri 'FIND_PACKAGE(Protobuf)' opencv-4.7.0 mysql-8.0.33 libphonenumber-8.13.11
opencv-4.7.0/cmake/OpenCVFindProtobuf.cmake:# BUILD_PROTOBUF=OFF: Custom manual protobuf configuration (see find_package(Protobuf) for details):
opencv-4.7.0/opencv_contrib-4.7.0/modules/cnn_3dobj/CMakeLists.txt:  find_package(Protobuf)
mysql-8.0.33/cmake/protobuf.cmake:    FIND_PACKAGE(Protobuf)
mysql-8.0.33/extra/protobuf/protobuf-3.19.4/cmake/examples.cmake:# sub_directory cannot be used because the find_package(protobuf) call would cause failures with redefined targets.
kloczek commented 1 year ago

BTW:

[tkloczko@pers-jacek protobuf]$ grep -r ifdef -h /usr/include/google/protobuf | sort | uniq
  #ifdef LIBPROTOBUF_EXPORTS
  #ifdef LIBPROTOC_EXPORTS
  #ifdef PROTOBUF_FORCE_COPY_DEFAULT_STRING
  #ifdef PROTOBUF_FORCE_COPY_IN_MOVE
  #ifdef PROTOBUF_FORCE_COPY_IN_SWAP
#  ifdef __SANITIZE_ADDRESS__
#  ifdef __SANITIZE_THREAD__
#ifdef ADDRESS_SANITIZER
#ifdef DEPRECATED_METHODS_TO_BE_DELETED
#ifdef GOOGLE_PROTOBUF_INTERNAL_DONATE_STEAL_INLINE
#ifdef GOOGLE_PROTOBUF_UTF8_VALIDATION_ENABLED
#ifdef NDEBUG
#ifdef PACKAGE
#ifdef PROTOBUF_ALIGNAS
#ifdef PROTOBUF_ALWAYS_INLINE
#ifdef PROTOBUF_ALWAYS_INLINE_CALL
#ifdef PROTOBUF_ASAN
#ifdef PROTOBUF_ASSUME
#ifdef PROTOBUF_ATTRIBUTE_INIT_PRIORITY1
#ifdef PROTOBUF_ATTRIBUTE_INIT_PRIORITY2
#ifdef PROTOBUF_ATTRIBUTE_NO_DESTROY
#ifdef PROTOBUF_ATTRIBUTE_REINITIALIZES
#ifdef PROTOBUF_ATTRIBUTE_STANDALONE_DEBUG
#ifdef PROTOBUF_ATTRIBUTE_WEAK
#ifdef PROTOBUF_BIG_ENDIAN
#ifdef PROTOBUF_BUILTIN_BSWAP16
#ifdef PROTOBUF_BUILTIN_BSWAP32
#ifdef PROTOBUF_BUILTIN_BSWAP64
#ifdef PROTOBUF_COLD
#ifdef PROTOBUF_CONSTINIT
#ifdef PROTOBUF_DID_UNDEF_LINUX
#ifdef PROTOBUF_DID_UNDEF_PACKAGE
#ifdef PROTOBUF_EXCLUSIVE_LOCKS_REQUIRED
#ifdef PROTOBUF_EXPORT
#ifdef PROTOBUF_FALLTHROUGH_INTENDED
#ifdef PROTOBUF_FIELD_OFFSET
#ifdef PROTOBUF_FINAL
#ifdef PROTOBUF_FORCE_ALLOCATION_ON_CONSTRUCTION
#ifdef PROTOBUF_FORCE_COPY_DEFAULT_STRING
#ifdef PROTOBUF_FORCE_COPY_IN_MOVE
#ifdef PROTOBUF_FORCE_COPY_IN_RELEASE
#ifdef PROTOBUF_FORCE_COPY_IN_SWAP
#ifdef PROTOBUF_FORCE_RESET_IN_CLEAR
#ifdef PROTOBUF_FUTURE_BREAKING_CHANGES
#ifdef PROTOBUF_FUZZ_MESSAGE_SPACE_USED_LONG
#ifdef PROTOBUF_GUARDED_BY
#ifdef PROTOBUF_LITTLE_ENDIAN
#ifdef PROTOBUF_LOCKS_EXCLUDED
#ifdef PROTOBUF_MIN_HEADER_VERSION_FOR_PROTOC
#ifdef PROTOBUF_MIN_PROTOC_VERSION
#ifdef PROTOBUF_MSAN
#ifdef PROTOBUF_MUSTTAIL
#ifdef PROTOBUF_NDEBUG_INLINE
#ifdef PROTOBUF_NODISCARD
#ifdef PROTOBUF_NOINLINE
#ifdef PROTOBUF_NO_THREADLOCAL
#ifdef PROTOBUF_NO_THREAD_SAFETY_ANALYSIS
#ifdef PROTOBUF_PRAGMA_INIT_SEG
#ifdef PROTOBUF_RESTRICT
#ifdef PROTOBUF_RETURNS_NONNULL
#ifdef PROTOBUF_RTTI
#ifdef PROTOBUF_SECTION_VARIABLE
#ifdef PROTOBUF_STABLE_EXPERIMENTS
#ifdef PROTOBUF_TAILCALL
#ifdef PROTOBUF_THREAD_LOCAL
#ifdef PROTOBUF_TSAN
#ifdef PROTOBUF_UNUSED
#ifdef PROTOBUF_USE_TABLE_PARSER_ON_REFLECTION
#ifdef PROTOBUF_VERSION
#ifdef PROTOBUF_VERSION_SUFFIX
#ifdef PROTOBUF_has_attribute_DEFINED_
#ifdef PROTOBUF_has_builtin_DEFINED_
#ifdef PROTOBUF_has_cpp_attribute_DEFINED_
#ifdef PROTOBUF_has_feature_DEFINED_
#ifdef PROTOBUF_has_warning_DEFINED_
#ifdef PROTOC_EXPORT
#ifdef SWIG
#ifdef _MSC_VER
#ifdef _WIN32
#ifdef _XBOX_ONE
#ifdef __APPLE__
#ifdef __GNUC__
#ifdef __aarch64__
#ifdef __clang__
#ifdef __cpp_aligned_new
#ifdef __cpp_if_constexpr
#ifdef linux
#ifdef major
#ifdef minor
// '#ifdef LANG_CXX11' to behave differently from '#if LANG_CXX11'.
// GCC, and MSVC. Function-like macros are usable without an #ifdef guard.
// use #ifdef the select the best implementation based on hardware / OS.

In first look most of those defines are used only on protobuf build time and whould not be used in public API definition.

mkruskal-google commented 1 year ago

Note: because you set BUILD_SHARED_LIBS=ON, I believe our cmake config will then enable protobuf_BUILD_SHARED_LIBS and PROTOBUF_USE_DLLS. That means protobuf will be built without ThreadSafeArena::thread_cache_. If your project then doesn't get PROTOBUF_USE_DLLS defined, you'd end up with a declaration of that field in thread_safe_arena.h but no linkage

mkruskal-google commented 1 year ago

I believe the issue is that you or your dependencies are using FindPackage, which doesn't properly set PROTOBUF_USE_DLLS as we would have expected from our installed .pc file. Your grep shows 2 places that are using FindPackage, and another with a somewhat ominous comment referencing it.

This would also explain your issues with utf8_range, since FindPackage doesn't know about our new dependencies

kloczek commented 1 year ago

Note: because you set BUILD_SHARED_LIBS=ON, I believe our cmake config will then enable protobuf_BUILD_SHARED_LIBS and PROTOBUF_USE_DLLS. That means protobuf will be built without ThreadSafeArena::thread_cache_. If your project then doesn't get PROTOBUF_USE_DLLS defined, you'd end up with a declaration of that field in thread_safe_arena.h but no linkage

Here is cmake -L result

-- Cache values
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr
GTest_DIR:PATH=/usr/lib64/cmake/GTest
absl_DIR:PATH=/usr/lib64/cmake/absl
protobuf_ABSL_PROVIDER:STRING=package
protobuf_ALLOW_CCACHE:BOOL=OFF
protobuf_BUILD_CONFORMANCE:BOOL=OFF
protobuf_BUILD_EXAMPLES:BOOL=OFF
protobuf_BUILD_LIBPROTOC:BOOL=OFF
protobuf_BUILD_PROTOBUF_BINARIES:BOOL=ON
protobuf_BUILD_PROTOC_BINARIES:BOOL=ON
protobuf_BUILD_SHARED_LIBS:BOOL=ON
protobuf_BUILD_TESTS:BOOL=ON
protobuf_DISABLE_RTTI:BOOL=OFF
protobuf_INSTALL:BOOL=ON
protobuf_INSTALL_EXAMPLES:BOOL=OFF
protobuf_REMOVE_INSTALLED_HEADERS:BOOL=OFF
protobuf_TEST_XML_OUTDIR:BOOL=OFF
protobuf_USE_EXTERNAL_GTEST:BOOL=ON
protobuf_WITH_ZLIB:BOOL=ON
utf8_range_ENABLE_INSTALL:BOOL=OFF
utf8_range_ENABLE_TESTS:BOOL=OFF

As you see on that list it is protobuf_BUILD_SHARED_LIBS:BOOL=ON and there is no PROTOBUF_USE_DLLS which should be set to some default (depends on what has been detected) value.

mkruskal-google commented 1 year ago

PROTOBUF_USE_DLLS isn't a cmake flag. We set it directly on our libraries via target_compile_definitions, only when protobuf_BUILD_SHARED_LIBS is set. So if you installed protobuf with protobuf_BUILD_SHARED_LIBS enabled, ThreadSafeArena::thread_cache_ won't exist. The fact that you're seeing linker errors about a missing symbol signals that someone is including our headers without PROTOBUF_USE_DLLS set. Since we put this directly on our cmake targets, this would only happen if you're not getting our cmake configs (e.g. if someone is using FIND_PACKAGE(Protobuf)).

As you showed in the original post, our .pc file does set PROTOBUF_USE_DLLS. So someone is using protobuf without that config

kloczek commented 1 year ago

PROTOBUF_USE_DLLS isn't a cmake flag. We set it directly on our libraries via target_compile_definitions, only when protobuf_BUILD_SHARED_LIBS is set. So if you installed protobuf with protobuf_BUILD_SHARED_LIBS enabled, ThreadSafeArena::thread_cache_ won't exist.

So looks like protobuf cmake is broken. Isn't it? 🤔

mkruskal-google commented 1 year ago

No, FIND_PACKAGE(Protobuf) is broken and both mysql-8.0.33 and opencv-4.7.0 are using it. It's causing our headers to see mismatched flags from what the libraries were built with

kloczek commented 1 year ago

As you showed in the original post, our .pc file does set PROTOBUF_USE_DLLS. So someone is using protobuf without that config

libpronenumber is not using cmake module and uses pkgconfig which in protobuf.pc has Cflags: -I${includedir} -DPROTOBUF_USE_DLLS and it fails on linking as well.

No, FIND_PACKAGE(Protobuf) is broken and both mysql-8.0.33 and opencv-4.7.0 are using it. It's causing our headers to see mismatched flags from what the libraries were built with

In both cases cmake is using installed by protobuf cmake module.

kloczek commented 1 year ago

Again: did you try to reproduce this issue using details which I've already provided? 🤔

kloczek commented 1 year ago 
[tkloczko@pers-jacek pkgconfig]$ grep -r PROTOBUF_USE_DLLS /usr/lib64/cmake/protobuf
/usr/lib64/cmake/protobuf/protobuf-targets.cmake:  INTERFACE_COMPILE_DEFINITIONS "PROTOBUF_USE_DLLS"
/usr/lib64/cmake/protobuf/protobuf-targets.cmake:  INTERFACE_COMPILE_DEFINITIONS "PROTOBUF_USE_DLLS"
/usr/lib64/cmake/protobuf/protobuf-targets.cmake:  INTERFACE_COMPILE_DEFINITIONS "PROTOBUF_USE_DLLS"
mkruskal-google commented 1 year ago

I have not seen a consistent enough set of reproduction steps to be able to reproduce this. You provided one set of cmake commands for building protobuf, but AFAICT it doesn't set protobuf_BUILD_SHARED_LIBS, which your other comments suggest you have set. It's also not clear what platform you're on, is this linux or windows?

mkruskal-google commented 1 year ago

Also, this utf8_range_ENABLE_INSTALL flag you're using is highly unsupported and I wouldn't expect it to work...

kloczek commented 1 year ago

Also, this utf8_range_ENABLE_INSTALL flag you're using is highly unsupported and I wouldn't expect it to work...

ItWorks™️ and is causes that utf8_range static library is not installed because it is used only on linking protobuf binaries.. As it is not part of public protobuf API/ABI it does not need to be installed.

kloczek commented 1 year ago

I have not seen a consistent enough set of reproduction steps to be able to reproduce this. You provided one set of cmake commands for building protobuf, but AFAICT it doesn't set protobuf_BUILD_SHARED_LIBS, which your other comments suggest you have set. It's also not clear what platform your on, is this linux or windows?

Linux and Solaris. In both cases all fails the same way. And please one more time have look on cmake -L output .. protobuf_BUILD_SHARED_LIBS:BOOL=ON is in that output. I did not set it but that with what end up cmake all because in main CMakeList.txt is

if (BUILD_SHARED_LIBS)
  set(protobuf_BUILD_SHARED_LIBS_DEFAULT ON)
else (BUILD_SHARED_LIBS)
  set(protobuf_BUILD_SHARED_LIBS_DEFAULT OFF)
endif (BUILD_SHARED_LIBS)

3rd time: did you try to reproduce that issue? 🤔

mkruskal-google commented 1 year ago

Here cmake parames which I have in my protobuf.spec

%build
%cmake \
       -D CMAKE_PREFIX_PATH=%{_prefix} \
       -D protobuf_ABSL_PROVIDER=ON \
       -D protobuf_ABSL_PROVIDER=package \
       -D protobuf_BUILD_TESTS=ON \
       -D protobuf_USE_EXTERNAL_GTEST=ON \
       -D utf8_range_ENABLE_INSTALL=OFF \
       %{nil}

This command doesn't have BUILD_SHARED_LIBS set either... Could you please provide reproduction steps that include all the flags you use to build:

I'm currently unable to build libpronenumber, because it seems stuck on C++11 which is incompatible with Abseil (it's ignoring -DCMAKE_CXX_STANDARD=14)

mkruskal-google commented 1 year ago

Manually fixing libphonenumber to C++14, I get the following error from their Abseil dependency:

/bin/ld: libphonenumber.a(phonenumber.pb.cc.o): undefined reference to symbol '_ZN4absl12lts_2023012512log_internal21CheckOpMessageBuilder7ForVar2Ev'
/bin/ld: /usr/local/lib/libabsl_log_internal_check_op.so.2301.0.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/geocoding_test_program.dir/build.make:141: geocoding_test_program] Error 1
make[1]: *** [CMakeFiles/Makefile2:410: CMakeFiles/geocoding_test_program.dir/all] Error 2

This looks similar to what you saw in mysql, but if I set -DBUILD_TESTING=OFF libphonenumber builds just fine. The commands I used were:

absl:

cmake . -DCMAKE_CXX_STANDARD=14 -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=ON
make install

protobuf:

cmake . -Dprotobuf_ABSL_PROVIDER=ON -Dprotobuf_ABSL_PROVIDER=package -Dprotobuf_BUILD_TESTS=OFF -Dprotobuf_USE_EXTERNAL_GTEST=ON -Dprotobuf_BUILD_SHARED_LIBS=ON
make install

libphonenumber:

cd cpp
cmake . -DCMAKE_PREFIX_PATH=/usr/local -DCMAKE_CXX_STANDARD=14 -DBUILD_TESTING=OFF
make