Open asfimport opened 3 years ago
Kouhei Sutou / @kou:
Could you show ls -lah /usr/lib/x86_64-linux-gnu/
?
❯ ls -lah /usr/lib/x86_64-linux-gnu/|grep -i zstd
-rw-r--r-- 1 root root 853K Mar 4 01:47 libzstd.a
lrwxrwxrwx 1 root root 16 Mar 4 01:47 libzstd.so -> libzstd.so.1.4.4
lrwxrwxrwx 1 root root 16 Mar 4 01:47 libzstd.so.1 -> libzstd.so.1.4.4
-rw-r--r-- 1 root root 671K Mar 4 01:47 libzstd.so.1.4.4
Nick Hortovanyi: Spent hours trying to solve this. Even tried from source. Same issue. I tried to not use zstd at all.
Installed Facebook's Folly
and have even tried to include this a submodule of my project
https://github.com/facebook/folly/blob/master/CMake/FindZstd.cmake
Presently worked out how to set ZSTD_LIB with ZSTD_LIBRARY
LIST(INSERT CMAKE_MODULE_PATH 0 ${CMAKE_CURRENT_SOURCE_DIR}/CMake)
find_package(Zstd REQUIRED)
if (ZSTD_FOUND)
set(ZSTD_LIB ${ZSTD_LIBRARY})
endif (ZSTD_FOUND
find_package(Arrow REQUIRED)
which works ... However now it cant find re2 and ares per the following ...
CMake Warning at /usr/local/lib/cmake/arrow/Findre2Alt.cmake:25 (find_package):
By not providing "Findre2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "re2", but
CMake did not find one.
Could not find a package configuration file provided by "re2" with any of
the following names:
re2Config.cmake
re2-config.cmake
Add the installation prefix of "re2" to CMAKE_PREFIX_PATH or set "re2_DIR"
to a directory containing one of the above files. If "re2" provides a
separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)
CMake Warning at /usr/local/lib/cmake/arrow/Findc-aresAlt.cmake:25 (find_package):
By not providing "Findc-ares.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "c-ares", but
CMake did not find one.
Could not find a package configuration file provided by "c-ares" with any
of the following names:
c-aresConfig.cmake
c-ares-config.cmake
Add the installation prefix of "c-ares" to CMAKE_PREFIX_PATH or set
"c-ares_DIR" to a directory containing one of the above files. If "c-ares"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)
Kouhei Sutou / @kou:
Does https://github.com/apache/arrow/tree/master/cpp/examples/minimal_build work with the official Ubuntu 20.04 libarrow-dev
package?
https://arrow.apache.org/install/
Nick Hortovanyi: You can try it with docker??
I've had this working previously. But havent touched the Apache Arrow C++ code for 6 months+
Kouhei Sutou / @kou: I did. It works without any problem.
Nick Hortovanyi: Only built this new server a couple of weeks ago
The cmake files that arrow is looking for, simply don’t exist.
They aren’t installed by the package maintainers as they do not provide them
Therefore I can’t build a cmake project against Apache Arrow for c++ on Ubuntu 20.04.2
I can install the libraries from the repo and/or via source. Just can’t build
Kouhei Sutou / @kou:
Could you create a Dockerfile
that reproduces this case?
Nick Hortovanyi: Just look at the packages I’ve mentioned. They don’t install CMake files
It’s self evident
Kouhei Sutou / @kou:
What are the packages you've mentioned? libarrow-dev
?
https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/focal/main/a/apache-arrow/libarrow-dev_4.0.1-1_amd64.deb
libzstd-dev
?
Nick Hortovanyi: I uninstalled those packages and installed from source https://arrow.apache.org/docs/developers/cpp/building.html
Nick Hortovanyi: dpkg -L libzstd-dev /. /usr /usr/include /usr/include/zbuff.h /usr/include/zdict.h /usr/include/zstd.h /usr/include/zstd_errors.h /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/libzstd.a /usr/lib/x86_64-linux-gnu/pkgconfig /usr/lib/x86_64-linux-gnu/pkgconfig/libzstd.pc /usr/share /usr/share/doc /usr/share/doc/libzstd-dev /usr/share/doc/libzstd-dev/README.Debian /usr/share/doc/libzstd-dev/copyright /usr/share/doc/libzstd-dev/examples /usr/share/doc/libzstd-dev/examples/Makefile /usr/share/doc/libzstd-dev/examples/README.md /usr/share/doc/libzstd-dev/examples/common.h /usr/share/doc/libzstd-dev/examples/dictionary_compression.c /usr/share/doc/libzstd-dev/examples/dictionary_decompression.c /usr/share/doc/libzstd-dev/examples/multiple_simple_compression.c /usr/share/doc/libzstd-dev/examples/multiple_streaming_compression.c /usr/share/doc/libzstd-dev/examples/simple_compression.c /usr/share/doc/libzstd-dev/examples/simple_decompression.c /usr/share/doc/libzstd-dev/examples/streaming_compression.c /usr/share/doc/libzstd-dev/examples/streaming_decompression.c /usr/share/doc/libzstd-dev/examples/streaming_memory_usage.c /usr/lib/x86_64-linux-gnu/libzstd.so /usr/share/doc/libzstd-dev/changelog.Debian.gz
Nick Hortovanyi: dpkg -s libzstd-dev Package: libzstd-dev Status: install ok installed Priority: optional Section: libdevel Installed-Size: 1079 Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com Architecture: amd64 Multi-Arch: same Source: libzstd Version: 1.4.4+dfsg-3ubuntu0.1 Depends: libzstd1 (= 1.4.4+dfsg-3ubuntu0.1) Description: fast lossless compression algorithm – development files Zstd, short for Zstandard, is a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level compression ratio. . This package contains the headers and static library. Homepage: https://github.com/facebook/zstd Original-Maintainer: Debian Med Packaging Team debian-med-packaging@lists.alioth.debian.org
Kouhei Sutou / @kou:
Our Findzstd.cmake
doesn't require CMake files for Zstandard: https://github.com/apache/arrow/blob/master/cpp/cmake_modules/Findzstd.cmake#L73-L75
So libzstd-dev
isn't a problem.
Installing pkg-config
may help you.
I can't help more unless you provide reproducible information such as Dockerfile
. Sorry.
Nick Hortovanyi: The Apache Arrow cmake files dont work with .pc files
pkg-config is installed
❯ dpkg -s pkg-config
Package: pkg-config
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 190
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Multi-Arch: foreign
Version: 0.29.1-0ubuntu4
Replaces: pkg-config-bin
Depends: libc6 (>= 2.14), libglib2.0-0 (>= 2.16.0), libdpkg-perl
Suggests: dpkg-dev
Conflicts: pkg-config-bin
Conffiles:
/etc/dpkg/dpkg.cfg.d/pkg-config-hook-config 467c877de00898418a9b26d432a1719a
Description: manage compile and link flags for libraries
pkg-config is a system for managing library compile and link flags that
works with automake and autoconf.
.
Increasingly libraries ship with ".pc" files that allow querying of the
compiler and linker flags needed to use them through the pkg-config(1)
program.
Original-Maintainer: Tollef Fog Heen <tfheen@debian.org>
Homepage: http://pkg-config.freedesktop.org
Nick Hortovanyi: Unfortunately I'm doing this inside of a ros2 galactic colcon build
You need ros2 galactic installed
Need this installed, built and sourced
https://github.com/aussierobots/ublox_dgnss
Followed by this project which is at the early stages. I'm trying to make sure the basics are compiling with colcon build
before writing the code:
https://github.com/aussierobots/ublox_ubx_arrow
As mentioned previously this type of project has built successfully before. The Apache
There appears to be a general issue not specific just with zstd now on ubuntu 20.04.2. Which is the current LTS version
Having found the workaround above for zstd. I'm now stuck with re2 and c-ares.
These errors are clearly coming from installed version of Apache Arrow from source now
CMake Warning at /usr/local/lib/cmake/arrow/Findre2Alt.cmake:25 (find_package):
By not providing "Findre2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "re2", but
CMake did not find one.
Could not find a package configuration file provided by "re2" with any of
the following names:
re2Config.cmake
re2-config.cmake
Add the installation prefix of "re2" to CMAKE_PREFIX_PATH or set "re2_DIR"
to a directory containing one of the above files. If "re2" provides a
separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:20 (find_package)
CMake Warning at /usr/local/lib/cmake/arrow/Findc-aresAlt.cmake:25 (find_package):
By not providing "Findc-ares.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "c-ares", but
CMake did not find one.
Could not find a package configuration file provided by "c-ares" with any
of the following names:
c-aresConfig.cmake
c-ares-config.cmake
Add the installation prefix of "c-ares" to CMAKE_PREFIX_PATH or set
"c-ares_DIR" to a directory containing one of the above files. If "c-ares"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:20 (find_package)
Nick Hortovanyi: I've fixed a different issue and am now able to compile with the two warnings mentioned above.
So if someone else is trying to create a workaround I'd suggest the zstd fix I mentioned above.
@kou is it possible that the original issue is that the Apache Arrow cmake is looking for ZSTD_LIB instead of ZSTD_LIBRARY?
Kouhei Sutou / @kou:
The variable name isn't related. ZSTD_LIB
should work.
We can't share reproducible instruction and you find a workaround. Can we close this issue?
Kouhei Sutou / @kou:
BTW, cmake ... -DCMAKE_FIND_DEBUG_MODE=ON
shows debug messages related to find_package()
. It may help us.
Nick Hortovanyi: I think the issue is, that arrow findzstd.cmake file does not work
Needs to be replaced with something simpler and similar to https://github.com/facebook/folly/blob/master/CMake/FindZstd.cmake
Kouhei Sutou / @kou:
If you think so, you can modify Findzstd.cmake
in your local environment and confirm whether the variable name is related or not.
I can't do it because I don't have an environment that reproduces this problem.
Nick Hortovanyi: I replaced /usr/local/lib/cmake/arrow/Findzstd.cmake with the Facebook Folly version.
CMake Error at CMakeLists.txt:19 (find_package):
Found package configuration file: /usr/local/lib/cmake/arrow/ArrowConfig.cmake but it set Arrow_FOUND to FALSE so package "Arrow" is considered to be NOT
FOUND. Reason given by package: Arrow could not be found because dependency zstd could not be found.
Kouhei Sutou / @kou:
Why did you use Folly's version to confirm whether the variable name in Findzstd.cmake
is related or not?
You should change Findzstd.cmake
and confirm. You don't need t o use Folly's version.
I thought that you said that we need the following change to fix this case:
diff --git a/cpp/cmake_modules/Findzstd.cmake b/cpp/cmake_modules/Findzstd.cmake
index 73b7ab250..82e1534e1 100644
--- a/cpp/cmake_modules/Findzstd.cmake
+++ b/cpp/cmake_modules/Findzstd.cmake
@@ -44,7 +44,7 @@ endif()
# First, find via if specified ZTD_ROOT
if(ZSTD_ROOT)
message(STATUS "Using ZSTD_ROOT: ${ZSTD_ROOT}")
- find_library(ZSTD_LIB
+ find_library(ZSTD_LIBRARY
NAMES ${ZSTD_LIB_NAMES}
PATHS ${ZSTD_ROOT}
PATH_SUFFIXES ${ARROW_LIBRARY_PATH_SUFFIXES}
@@ -63,14 +63,14 @@ else()
set(ZSTD_INCLUDE_DIR "${ZSTD_PC_INCLUDEDIR}")
list(APPEND ZSTD_PC_LIBRARY_DIRS "${ZSTD_PC_LIBDIR}")
- find_library(ZSTD_LIB
+ find_library(ZSTD_LIBRARY
NAMES ${ZSTD_LIB_NAMES}
PATHS ${ZSTD_PC_LIBRARY_DIRS}
NO_DEFAULT_PATH
PATH_SUFFIXES ${ARROW_LIBRARY_PATH_SUFFIXES})
else()
# Third, check all other CMake paths
- find_library(ZSTD_LIB
+ find_library(ZSTD_LIBRARY
NAMES ${ZSTD_LIB_NAMES}
PATH_SUFFIXES ${ARROW_LIBRARY_PATH_SUFFIXES})
find_path(ZSTD_INCLUDE_DIR
@@ -79,11 +79,11 @@ else()
endif()
endif()
-find_package_handle_standard_args(zstd REQUIRED_VARS ZSTD_LIB ZSTD_INCLUDE_DIR)
+find_package_handle_standard_args(zstd REQUIRED_VARS ZSTD_LIBRARY ZSTD_INCLUDE_DIR)
if(zstd_FOUND)
add_library(zstd::libzstd UNKNOWN IMPORTED)
set_target_properties(zstd::libzstd
- PROPERTIES IMPORTED_LOCATION "${ZSTD_LIB}"
+ PROPERTIES IMPORTED_LOCATION "${ZSTD_LIBRARY}"
INTERFACE_INCLUDE_DIRECTORIES "${ZSTD_INCLUDE_DIR}")
endif()
If you want to use Folly's version, can we close this?
Nick Hortovanyi: The arrow version doesn’t work. It’s overly complex.
The Facebook folly version works. And I assume as well on most platforms
I don’t understand how Findarrow.cmake interacts with Findzstd.cmake. I don’t have the time to work that out.
My work around, is just that, a work around. I didn’t need to do it a few months ago.
Basically when I built it I enabled plasma, compute, flight, parquet, cuda and Python. I did not enable hive, Hadoop, gandiva.
When I tried it with prebuilt packages, I installed all libraries.
As you’ve confirmed there is nothing wrong with how zstd is installed.
Nick Hortovanyi: tried your suggestion above
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
Could NOT find zstd (missing: ZSTD_LIBRARY)
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
/usr/local/lib/cmake/arrow/Findzstd.cmake:85 (find_package_handle_standard_args)
/usr/share/cmake-3.16/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
/usr/local/lib/cmake/arrow/ArrowConfig.cmake:86 (find_dependency)
CMakeLists.txt:19 (find_package)
Kouhei Sutou / @kou: It shows that the variable name isn't related.
Do you have any more tries you want to do in this issue? cmake ... -DCMAKE_FIND_DEBUG_MODE=ON
?
Can we close this issue?
Nick Hortovanyi: I experimented on another machine last night. That machine had been upgraded over the years probably from Ubuntu 16.04 -> Ubuntu 18.04 -> Ubuntu 20.04 and the cmake-gui configuration messages are a lot more detailed with pkg-config messages. However I didnt complete a full compile (long story - as I had protobufs installed locally which needed upgrading and other packages).
This issue is clearly related to a fresh install of Ubuntu 20.04.2 with the versions of clang, cmake etc installed.
I'd also suggest its pointless looking for Findzstd.cmake, FindRE2.cmake and FindC-Ares.cmake in the Arrow cmake modules. They will never exist as the package maintainers don't produce them.
So no I dont think this issue can be closed. The arrow cmake files just dont make sense and/or instructions are required on what to install in retro.
I dont have the time to work it all out at the moment.
Kouhei Sutou / @kou: Summary:
cpp/examples/minimal_build
on Ubuntu 20.04 doesn't have this problem.
cpp/examples/minimal_build
isn't tried on the environment.-DCMAKE_FIND_DEBUG_MODE=ON
isn't tried on the environment.[~horto]
is the only person that can confirm this for now.
[~horto]
doesn't have the time to work on this for now.We don't have the next action to solve this problem for now.
I'm unable to use the pre-built c++ arrow libraries with a project using a CMakeList.txt containing
Giving the following error
libzstd and libstd-dev are both installed
Environment: Ubuntu 20.04.2 Reporter: Nick Hortovanyi
Note: This issue was originally created as ARROW-13334. Please see the migration documentation for further details.