search

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.48k stars 3.52k forks source link

Findzstd.cmake doesnt find zstd on Ubuntu 20.04 #18742

Open asfimport opened 3 years ago

asfimport commented 3 years ago

I'm unable to use the pre-built c++ arrow libraries with a project using a CMakeList.txt containing


find_package(Arrow REQUIRED)

Giving the following error


CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
Could NOT find zstd (missing: ZSTD_LIB)
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
/usr/lib/x86_64-linux-gnu/cmake/arrow/Findzstd.cmake:82 (find_package_handle_standard_args)
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/lib/x86_64-linux-gnu/cmake/arrow/ArrowConfig.cmake:96 (find_dependency)
CMakeLists.txt:12 (find_package)

libzstd and libstd-dev are both installed

 

Environment: Ubuntu 20.04.2 Reporter: Nick Hortovanyi

Note: This issue was originally created as ARROW-13334. Please see the migration documentation for further details.

asfimport commented 3 years ago

Kouhei Sutou / @kou: Could you show ls -lah /usr/lib/x86_64-linux-gnu/?

asfimport commented 3 years ago

Nick Hortovanyi:


❯ ls -lah /usr/lib/x86_64-linux-gnu/|grep -i zstd
 -rw-r--r-- 1 root root 853K Mar 4 01:47 libzstd.a
 lrwxrwxrwx 1 root root 16 Mar 4 01:47 libzstd.so -> libzstd.so.1.4.4
 lrwxrwxrwx 1 root root 16 Mar 4 01:47 libzstd.so.1 -> libzstd.so.1.4.4
 -rw-r--r-- 1 root root 671K Mar 4 01:47 libzstd.so.1.4.4
asfimport commented 3 years ago

Nick Hortovanyi: Spent hours trying to solve this. Even tried from source. Same issue. I tried to not use zstd at all.

Installed Facebook's Folly

and have even tried to include this a submodule of my project

https://github.com/facebook/folly/blob/master/CMake/FindZstd.cmake

 

Presently worked out how to set ZSTD_LIB with ZSTD_LIBRARY


LIST(INSERT CMAKE_MODULE_PATH 0 ${CMAKE_CURRENT_SOURCE_DIR}/CMake)
find_package(Zstd REQUIRED)
if (ZSTD_FOUND)
  set(ZSTD_LIB ${ZSTD_LIBRARY})
endif (ZSTD_FOUND
find_package(Arrow REQUIRED)

which works ... However now it cant find re2 and ares per the following ...


CMake Warning at /usr/local/lib/cmake/arrow/Findre2Alt.cmake:25 (find_package):
By not providing "Findre2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "re2", but
CMake did not find one.
 
Could not find a package configuration file provided by "re2" with any of
the following names:
 
re2Config.cmake
re2-config.cmake
 
Add the installation prefix of "re2" to CMAKE_PREFIX_PATH or set "re2_DIR"
to a directory containing one of the above files. If "re2" provides a
separate development package or SDK, be sure it has been installed.

Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)
 
 
CMake Warning at /usr/local/lib/cmake/arrow/Findc-aresAlt.cmake:25 (find_package):
By not providing "Findc-ares.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "c-ares", but
CMake did not find one.
 
Could not find a package configuration file provided by "c-ares" with any
of the following names:
 
c-aresConfig.cmake
c-ares-config.cmake
 
Add the installation prefix of "c-ares" to CMAKE_PREFIX_PATH or set
"c-ares_DIR" to a directory containing one of the above files. If "c-ares"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)
asfimport commented 3 years ago

Kouhei Sutou / @kou: Does https://github.com/apache/arrow/tree/master/cpp/examples/minimal_build work with the official Ubuntu 20.04 libarrow-dev package? https://arrow.apache.org/install/

asfimport commented 3 years ago

Nick Hortovanyi: You can try it with docker?? 

 

I've had this working previously. But havent touched the Apache Arrow C++ code for 6 months+ 

asfimport commented 3 years ago

Kouhei Sutou / @kou: I did. It works without any problem.

asfimport commented 3 years ago

Nick Hortovanyi: Only built this new server a couple of weeks ago

The cmake files that arrow is looking for, simply don’t exist.

They aren’t installed by the package maintainers as they do not provide them

Therefore I can’t build a cmake project against Apache Arrow for c++ on Ubuntu 20.04.2

I can install the libraries from the repo and/or via source. Just can’t build

asfimport commented 3 years ago

Kouhei Sutou / @kou: Could you create a Dockerfile that reproduces this case?

asfimport commented 3 years ago

Nick Hortovanyi: Just look at the packages I’ve mentioned. They don’t install CMake files

It’s self evident

asfimport commented 3 years ago

Kouhei Sutou / @kou: What are the packages you've mentioned? libarrow-dev? https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/focal/main/a/apache-arrow/libarrow-dev_4.0.1-1_amd64.deb

libzstd-dev?

asfimport commented 3 years ago

Nick Hortovanyi: I uninstalled those packages and installed from source https://arrow.apache.org/docs/developers/cpp/building.html

asfimport commented 3 years ago

Nick Hortovanyi: dpkg -L libzstd-dev /. /usr /usr/include /usr/include/zbuff.h /usr/include/zdict.h /usr/include/zstd.h /usr/include/zstd_errors.h /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/libzstd.a /usr/lib/x86_64-linux-gnu/pkgconfig /usr/lib/x86_64-linux-gnu/pkgconfig/libzstd.pc /usr/share /usr/share/doc /usr/share/doc/libzstd-dev /usr/share/doc/libzstd-dev/README.Debian /usr/share/doc/libzstd-dev/copyright /usr/share/doc/libzstd-dev/examples /usr/share/doc/libzstd-dev/examples/Makefile /usr/share/doc/libzstd-dev/examples/README.md /usr/share/doc/libzstd-dev/examples/common.h /usr/share/doc/libzstd-dev/examples/dictionary_compression.c /usr/share/doc/libzstd-dev/examples/dictionary_decompression.c /usr/share/doc/libzstd-dev/examples/multiple_simple_compression.c /usr/share/doc/libzstd-dev/examples/multiple_streaming_compression.c /usr/share/doc/libzstd-dev/examples/simple_compression.c /usr/share/doc/libzstd-dev/examples/simple_decompression.c /usr/share/doc/libzstd-dev/examples/streaming_compression.c /usr/share/doc/libzstd-dev/examples/streaming_decompression.c /usr/share/doc/libzstd-dev/examples/streaming_memory_usage.c /usr/lib/x86_64-linux-gnu/libzstd.so /usr/share/doc/libzstd-dev/changelog.Debian.gz

asfimport commented 3 years ago

Nick Hortovanyi: dpkg -s libzstd-dev Package: libzstd-dev Status: install ok installed Priority: optional Section: libdevel Installed-Size: 1079 Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com Architecture: amd64 Multi-Arch: same Source: libzstd Version: 1.4.4+dfsg-3ubuntu0.1 Depends: libzstd1 (= 1.4.4+dfsg-3ubuntu0.1) Description: fast lossless compression algorithm – development files Zstd, short for Zstandard, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level compression ratio. . This package contains the headers and static library. Homepage: https://github.com/facebook/zstd Original-Maintainer: Debian Med Packaging Team debian-med-packaging@lists.alioth.debian.org

asfimport commented 3 years ago

Kouhei Sutou / @kou: Our Findzstd.cmake doesn't require CMake files for Zstandard: https://github.com/apache/arrow/blob/master/cpp/cmake_modules/Findzstd.cmake#L73-L75 So libzstd-dev isn't a problem.

Installing pkg-config may help you.

I can't help more unless you provide reproducible information such as Dockerfile. Sorry.

asfimport commented 3 years ago

Nick Hortovanyi: The Apache Arrow cmake files dont work with .pc files

pkg-config is installed

 
❯ dpkg -s pkg-config
Package: pkg-config
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 190
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Multi-Arch: foreign
Version: 0.29.1-0ubuntu4
Replaces: pkg-config-bin
Depends: libc6 (>= 2.14), libglib2.0-0 (>= 2.16.0), libdpkg-perl
Suggests: dpkg-dev
Conflicts: pkg-config-bin
Conffiles:
/etc/dpkg/dpkg.cfg.d/pkg-config-hook-config 467c877de00898418a9b26d432a1719a
Description: manage compile and link flags for libraries
pkg-config is a system for managing library compile and link flags that
works with automake and autoconf.
.
Increasingly libraries ship with ".pc" files that allow querying of the
compiler and linker flags needed to use them through the pkg-config(1)
program.
Original-Maintainer: Tollef Fog Heen <tfheen@debian.org>
Homepage: http://pkg-config.freedesktop.org
asfimport commented 3 years ago

Nick Hortovanyi: Unfortunately I'm doing this inside of a ros2 galactic colcon build

 

You need ros2 galactic installed

 

Need this installed, built and sourced

https://github.com/aussierobots/ublox_dgnss

 

Followed by this project which is at the early stages. I'm trying to make sure the basics are compiling with colcon build before writing the code:

https://github.com/aussierobots/ublox_ubx_arrow

 

As mentioned previously this type of project has built successfully before. The Apache 

There appears to be a general issue not specific just with zstd now on ubuntu 20.04.2. Which is the current LTS version

 

Having found the workaround above for zstd. I'm now stuck with re2 and c-ares.

 

These errors are clearly coming from installed version of Apache Arrow from source now


CMake Warning at /usr/local/lib/cmake/arrow/Findre2Alt.cmake:25 (find_package):
By not providing "Findre2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "re2", but
CMake did not find one.
 
Could not find a package configuration file provided by "re2" with any of
the following names:
 
re2Config.cmake
re2-config.cmake
 
Add the installation prefix of "re2" to CMAKE_PREFIX_PATH or set "re2_DIR"
to a directory containing one of the above files. If "re2" provides a
separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:20 (find_package)
 
 
CMake Warning at /usr/local/lib/cmake/arrow/Findc-aresAlt.cmake:25 (find_package):
By not providing "Findc-ares.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "c-ares", but
CMake did not find one.
 
Could not find a package configuration file provided by "c-ares" with any
of the following names:
 
c-aresConfig.cmake
c-ares-config.cmake
 
Add the installation prefix of "c-ares" to CMAKE_PREFIX_PATH or set
"c-ares_DIR" to a directory containing one of the above files. If "c-ares"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:20 (find_package)
asfimport commented 3 years ago

Nick Hortovanyi: I've fixed a different issue and am now able to compile with the two warnings mentioned above. 

So if someone else is trying to create a workaround I'd suggest the zstd fix I mentioned above.

 

@kou  is it possible that the original issue is that the Apache Arrow cmake is looking for ZSTD_LIB instead of ZSTD_LIBRARY?

asfimport commented 3 years ago

Kouhei Sutou / @kou: The variable name isn't related. ZSTD_LIB should work.

We can't share reproducible instruction and you find a workaround. Can we close this issue?

asfimport commented 3 years ago

Kouhei Sutou / @kou: BTW, cmake ... -DCMAKE_FIND_DEBUG_MODE=ON shows debug messages related to find_package(). It may help us.

asfimport commented 3 years ago

Nick Hortovanyi: I think the issue is, that arrow findzstd.cmake file does not work

Needs to be replaced with something simpler and similar to https://github.com/facebook/folly/blob/master/CMake/FindZstd.cmake

asfimport commented 3 years ago

Kouhei Sutou / @kou: If you think so, you can modify Findzstd.cmake in your local environment and confirm whether the variable name is related or not. I can't do it because I don't have an environment that reproduces this problem.

asfimport commented 3 years ago

Nick Hortovanyi: I replaced /usr/local/lib/cmake/arrow/Findzstd.cmake with the Facebook Folly version.

 


CMake Error at CMakeLists.txt:19 (find_package):                                                                                                               
Found package configuration file:                                                                                                                                                                                                                                                                                           /usr/local/lib/cmake/arrow/ArrowConfig.cmake                                                                                                                                                                                                                                                                            but it set Arrow_FOUND to FALSE so package "Arrow" is considered to be NOT                                                                                   
FOUND.  Reason given by package:                                                                                                                                                                                                                                                                                          Arrow could not be found because dependency zstd could not be found.

 

asfimport commented 3 years ago

Nick Hortovanyi: My fix in here works

https://github.com/aussierobots/ublox_ubx_arrow/blob/main/ublox_ubx_arrow_sink_node/CMakeLists.txt

 

asfimport commented 3 years ago

Kouhei Sutou / @kou: Why did you use Folly's version to confirm whether the variable name in Findzstd.cmake is related or not? You should change Findzstd.cmake and confirm. You don't need t o use Folly's version.

I thought that you said that we need the following change to fix this case:


diff --git a/cpp/cmake_modules/Findzstd.cmake b/cpp/cmake_modules/Findzstd.cmake
index 73b7ab250..82e1534e1 100644
--- a/cpp/cmake_modules/Findzstd.cmake
+++ b/cpp/cmake_modules/Findzstd.cmake
@@ -44,7 +44,7 @@ endif()
# First, find via if specified ZTD_ROOT
 if(ZSTD_ROOT)
   message(STATUS "Using ZSTD_ROOT: ${ZSTD_ROOT}")
-  find_library(ZSTD_LIB
+  find_library(ZSTD_LIBRARY
                NAMES ${ZSTD_LIB_NAMES}
                PATHS ${ZSTD_ROOT}
                PATH_SUFFIXES ${ARROW_LIBRARY_PATH_SUFFIXES}
@@ -63,14 +63,14 @@ else()
     set(ZSTD_INCLUDE_DIR "${ZSTD_PC_INCLUDEDIR}")

     list(APPEND ZSTD_PC_LIBRARY_DIRS "${ZSTD_PC_LIBDIR}")
-    find_library(ZSTD_LIB
+    find_library(ZSTD_LIBRARY
                  NAMES ${ZSTD_LIB_NAMES}
                  PATHS ${ZSTD_PC_LIBRARY_DIRS}
                  NO_DEFAULT_PATH
                  PATH_SUFFIXES ${ARROW_LIBRARY_PATH_SUFFIXES})
   else()
     # Third, check all other CMake paths
-    find_library(ZSTD_LIB
+    find_library(ZSTD_LIBRARY
                  NAMES ${ZSTD_LIB_NAMES}
                  PATH_SUFFIXES ${ARROW_LIBRARY_PATH_SUFFIXES})
     find_path(ZSTD_INCLUDE_DIR
@@ -79,11 +79,11 @@ else()
   endif()
 endif()

-find_package_handle_standard_args(zstd REQUIRED_VARS ZSTD_LIB ZSTD_INCLUDE_DIR)
+find_package_handle_standard_args(zstd REQUIRED_VARS ZSTD_LIBRARY ZSTD_INCLUDE_DIR)

 if(zstd_FOUND)
   add_library(zstd::libzstd UNKNOWN IMPORTED)
   set_target_properties(zstd::libzstd
-                        PROPERTIES IMPORTED_LOCATION "${ZSTD_LIB}"
+                        PROPERTIES IMPORTED_LOCATION "${ZSTD_LIBRARY}"
                                    INTERFACE_INCLUDE_DIRECTORIES "${ZSTD_INCLUDE_DIR}")
 endif()

If you want to use Folly's version, can we close this?

asfimport commented 3 years ago

Nick Hortovanyi: The arrow version doesn’t work. It’s overly complex.

The Facebook folly version works. And I assume as well on most platforms

I don’t understand how Findarrow.cmake interacts with Findzstd.cmake. I don’t have the time to work that out.

My work around, is just that, a work around. I didn’t need to do it a few months ago.

Basically when I built it I enabled plasma, compute, flight, parquet, cuda and Python. I did not enable hive, Hadoop, gandiva.

When I tried it with prebuilt packages, I installed all libraries.

As you’ve confirmed there is nothing wrong with how zstd is installed.

asfimport commented 3 years ago

Nick Hortovanyi: tried your suggestion above 


CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message): 
Could NOT find zstd (missing: ZSTD_LIBRARY) 
Call Stack (most recent call first):
 /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
 /usr/local/lib/cmake/arrow/Findzstd.cmake:85 (find_package_handle_standard_args)
 /usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
 /usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
 CMakeLists.txt:19 (find_package)
asfimport commented 3 years ago

Kouhei Sutou / @kou: It shows that the variable name isn't related.

Do you have any more tries you want to do in this issue? cmake ... -DCMAKE_FIND_DEBUG_MODE=ON?

Can we close this issue?

asfimport commented 3 years ago

Nick Hortovanyi: I experimented on another machine last night. That machine had been upgraded over the years probably from Ubuntu 16.04 -> Ubuntu 18.04 -> Ubuntu 20.04 and the cmake-gui configuration messages are a lot more detailed with pkg-config messages. However I didnt complete a full compile (long story - as I had protobufs installed locally which needed upgrading and other packages).

This issue is clearly related to a fresh install of Ubuntu 20.04.2 with the versions of clang, cmake etc installed.

I'd also suggest its pointless looking for Findzstd.cmake, FindRE2.cmake and FindC-Ares.cmake in the Arrow cmake modules. They will never exist as the package maintainers don't produce them.

So no I dont think this issue can be closed. The arrow cmake files just dont make sense and/or instructions are required on what to install in retro.

I dont have the time to work it all out at the moment.

asfimport commented 3 years ago

Kouhei Sutou / @kou: Summary: