Closed 3unjee closed 2 years ago
perhaps this is an issue where try-signal
doesn't support android x86 very well. do you see the try_signal
library being built and linked against?
I think this happens because you build libtorrent as a static library. Static libraries (typically) don't embed all of their dependencies in them (since they're really just archives of object files). Instead, static libraries punt on linking against all their dependencies to its dependee. i.e. you are expected to link against the other static libraries that libtorrent depends on.
Figuring this out and doing it correctly is the job of the build system. If you were to create your shared library (that links against libtorrent) using boost-build (it might work with cmake too), I would expect the dependency to be propagated correctly and the final .so
would include all necessary static libraries.
perhaps this is an issue where
try-signal
doesn't support android x86 very well. do you see thetry_signal
library being built and linked against?
I can confirm try_signal is strangely never built when I'm passing static to b2, could it be an issue with the Jamfile ? I'm seeing this on Android armv8 and x86_64.
I temporarly rolled back to libtorrent 1.2.13 which builds everywhere for me with C++14.
I think this happens because you build libtorrent as a static library. Static libraries (typically) don't embed all of their dependencies in them (since they're really just archives of object files). Instead, static libraries punt on linking against all their dependencies to its dependee. i.e. you are expected to link against the other static libraries that libtorrent depends on.
Figuring this out and doing it correctly is the job of the build system. If you were to create your shared library (that links against libtorrent) using boost-build (it might work with cmake too), I would expect the dependency to be propagated correctly and the final
.so
would include all necessary static libraries.
So would this mean I'd have to build try_signal separatly and link statically against it while building my own shared library ?
Also FYI, for Android it seems I had to move to NDK 29 to support C++14 std library fully.
I can confirm try_signal is strangely never built when I'm passing static to b2, could it be an issue with the Jamfile ? I'm seeing this on Android armv8 and x86_64.
That does seem to be an issue. I'll look into that. Perhaps b2 builds this lazily, and since it doesn't see an actual binary (executable or shared library) depend on it, it won't build it.
Seems the issue is generated libtorrent-rasterbar.pc
by b2
.
Here is the b2
generated libtorrent-rasterbar.pc
:
$ cat /usr/share/pkgconfig/libtorrent-rasterbar.pc
Name: libtorrent-rasterbar
Description: libtorrent is an open source C++ library implementing the BitTorrent protocol
URL: https://libtorrent.org
Version: 2.0.3
Libs: -L"/usr/lib" -L"/opt/homebrew/lib" -L"/usr/local/opt/boost/lib" -ltorrent-rasterbar -lboost_system -lcrypto -ldl -lssl
Libs.private: -L"/usr/lib" -L"/opt/homebrew/lib" -L"/usr/local/opt/boost/lib" -llibtry_signal.a
Cflags: -DTORRENT_USE_LIBCRYPTO -DTORRENT_USE_OPENSSL -DTORRENT_SSL_PEERS -DTORRENT_USE_I2P=1 -DBOOST_ALL_NO_LIB -DBOOST_ASIO_ENABLE_CANCELIO -DBOOST_ASIO_HAS_STD_CHRONO -DBOOST_ASIO_NO_DEPRECATED -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION -DBOOST_SYSTEM_NO_DEPRECATED -DTORRENT_LINKING_SHARED -I"/usr/include" -I"/opt/homebrew/include" -I"/usr/local/opt/boost/include"
But by cmake
:
$ cat /usr/lib/pkgconfig/libtorrent-rasterbar.pc
prefix=/usr
libdir=${prefix}/lib
Name: libtorrent
Description: Bittorrent library
Version: 2.0.3
Libs: -L${libdir} -ltorrent-rasterbar -lpthread -lssl -lcrypto
Cflags: -fexceptions -I/usr/include -DTORRENT_LINKING_SHARED -DBOOST_ASIO_ENABLE_CANCELIO -DBOOST_ASIO_NO_DEPRECATED -DTORRENT_USE_OPENSSL -DTORRENT_USE_LIBCRYPTO -DTORRENT_SSL_PEERS -DOPENSSL_NO_SSL2
First, the location of pkgconfig
seems wrong, The correct way should be ${prefix}/lib/pkgconfig/
, so cmake
is right.
Second, the libtry_signal.a
is private dependency in b2
, but cannot find in cmake
, maybe the libtry_signal
is static linked in cmake
?
To temporary solved this issue, you may add libtry_signal
to linked libs by youself. E.g:
clang++ -ltry_signal ....
First, the location of pkgconfig seems wrong, The correct way should be ${prefix}/lib/pkgconfig/, so cmake is right.
Do you have a link to documentation about that?
When I was implementing this in the Jamfile
I got the impression both of those paths are used.
Second, the libtry_signal.a is private dependency in b2, but cannot find in cmake, maybe the libtry_signal is static linked in cmake?
It's unconditionally statically linked in the Jamfile
. Is a private dependency the wrong kind of dependency? I figured since its API isn't exposed as part of libtorrent's, it should be private.
First, the location of pkgconfig seems wrong, The correct way should be ${prefix}/lib/pkgconfig/, so cmake is right.
Do you have a link to documentation about that? When I was implementing this in the
Jamfile
I got the impression both of those paths are used.Second, the libtry_signal.a is private dependency in b2, but cannot find in cmake, maybe the libtry_signal is static linked in cmake?
It's unconditionally statically linked in the
Jamfile
. Is a private dependency the wrong kind of dependency? I figured since its API isn't exposed as part of libtorrent's, it should be private.
https://superuser.com/questions/47967/where-does-pkg-config-takes-its-information-from. In most cases, /usr/lib/pkgconfig
is prefer, and most programs will install pkgconfig files into ${prefix}/lib/pkgconfig/
. So I think should use ${prefer}/lib/pkgconfig
first.
And here is my qbittorrent-enhanced-edition build script: https://github.com/abcfy2/qBittorrent-Enhanced-Edition/blob/v4_3_x/.github/workflows/cross_build.sh#L206
I must add LIBS='-ltry_signal'
to configure
script, otherwise will cause this issue.
I must add
LIBS='-ltry_signal'
toconfigure
script, otherwise will cause this issue.
This seems to be what's missing in my Android static build of 2.0.x.
Here is my build log: https://dev.azure.com/bunjee/libtorrent/_build/results?buildId=3107&view=logs&j=189e2a2f-bff7-5384-4f13-0343cadba808&t=d2e68086-b926-544e-9d8d-06b86f258040
@arvidn I can confirm lib/pkgconfig/libtorrent-rasterbar.pc
is the right place, especially for cmake
. If i am using
-D CMAKE_INSTALL_PREFIX="${qb_install_dir}"
Which translates into
-D CMAKE_INSTALL_PREFIX="/root/qb-build"
The cmake is ignoring the share/pkgconfig/libtorrent-rasterbar.pc
and only loading my custom on in /root/qb-build/lib/pkgconfig/libtorrent-rasterbar.pc
@3unjee You can make your own lib/pkgconfig/libtorrent-rasterbar.pc
as discussed here https://github.com/arvidn/libtorrent/discussions/6211
Otherwise when you build with b2
no pkgconfig is created (in the right place) and programs have no way to know libtry_signal.a
exists. So you can create a pkgconfig
file like this
prefix=/root/qb-build
libdir=${exec_prefix}/lib
includedir=${prefix}/include
boostinclude=${prefix}/boost
Name: libtorrent-rasterbar
Description: The libtorrent-rasterbar library
Version: 2.0.3.0
Requires:
Libs: -L${libdir} -l:libtorrent-rasterbar.a -l:libtry_signal.a
Cflags: -I${includedir} -I${boostinclude}
OR pass it using LDFLAGS="-l:libtrysignal.a"
or LIBS="-l:libtrysignal.a"
@userdocs Couldn't libtrysignal
be contained inside of libtorrent ?
b2
creates them but if it can be unified is a question for @arvid . I suppose it's like libcrypto.a
and libssl.a
. You need both but they are not a unified library and I guess there is a reason.
When i build libtorrent statically i get this.
v1_2 = libboost_system.a
+ libtorrent-rasterbar.a
v2_0 = libboost_system.a
+ libtorrent-rasterbar.a
+ libtrysignal.a
Okay, just checked I have a libtry_signal.a
in my deps/try_signal
folder indeed. So I guess I have to link against this too.
It seems that it's not required when using libtorrent 2.0.x as a shared library though, can anybody explain why ?
Well I think @arvidn kinda explained it here already:
I think this happens because you build libtorrent as a static library. Static libraries (typically) don't embed all of their dependencies in them (since they're really just archives of object files). Instead, static libraries punt on linking against all their dependencies to its dependee. i.e. you are expected to link against the other static libraries that libtorrent depends on.
Still, that's strange I didn't see anything mentioning try_signal
in the build log.
This will show you how it works for static by installing it to the $HOME/lt-build/libs
.
curl -sNLk https://boostorg.jfrog.io/artifactory/main/release/1.76.0/source/boost_1_76_0.tar.gz --create-dirs -o "$HOME/lt-build/boost_1_76_0.tar.gz"
tar xf "$HOME/lt-build/boost_1_76_0.tar.gz" -C "$HOME/lt-build"
pushd "$HOME/lt-build/boost_1_76_0/" || exit
"$HOME/lt-build/boost_1_76_0/bootstrap.sh"
#
export BOOST_BUILD_PATH="$HOME/lt-build/boost_1_76_0"
#
echo "using gcc : $(gcc -dumpversion) : g++-$(g++ -dumpversion) ;" > "$HOME/user-config.jam" # -dumpversion may give return a bad result outside a debian based systems
#
git clone --single-branch --branch RC_2_0 --shallow-submodules --recurse-submodules --depth 1 https://github.com/arvidn/libtorrent "$HOME/lt-build/libtorrent"
#
pushd "$HOME/lt-build/libtorrent" || exit
"$HOME/lt-build/boost_1_76_0/b2" -j"$(nproc)" optimization=speed crypto=openssl cxxstd=17 variant=release threading=multi link=static boost-link=static install --prefix="$HOME/lt-build/libs"
If you also wanted the python bindings then you just add this
# You also want the python bindings?
python_major="$(python3 -c "import sys; print(sys.version_info[0])")"
python_minor="$(python3 -c "import sys; print(sys.version_info[1])")"
#
echo "using python : ${python_major}.${python_minor} : /usr/bin/python${python_major}.${python_minor} : /usr/include/python${python_major}.${python_minor} : /usr/lib/python${python_major}.${python_minor} ;" >> "$HOME/user-config.jam"
#
pushd "$HOME/lt-build/libtorrent/bindings/python" || exit
#
"$HOME/lt-build/boost_1_76_0/b2" -j"$(nproc)" optimization=speed crypto=openssl cxxstd=17 variant=release threading=multi libtorrent-link=static boost-link=static install_module python-install-path="$HOME/lt-build/libs"
Thanks for sharing, is the optimization=speed
a significant improvement ?
I am going to take a guess here as say it more to do with gcc
optimisations. See here https://gcc.gnu.org/onlinedocs/gnat_ugn/Optimization-Levels.html
Significant improvement is too subject to answer.
I think with boost you have optimization=speed
and optimization=size
and if you don't set it one may get defaulted to. Probably size.
This probably translates into -O2
/ -Os
or -O3
@arvidn @userdocs Here is a quick recap on the situations.
I'm successfully building libtorrent 2.0.3 on Android on all four platforms with the following:
b2 clang-arm -j4 cxxflags="-fPIC -DANDROID" cxxstd=14 variant=release link=static threading=multi
Unfortunately when I try to link against libtorrent-rasterbar.a
I get missing symbols from the try_signal
dependency. Turns out that dependency is never built.
That didn't stop me from building it manually with the very same b2 command. Unfortunately, that fails because test.cpp
and example.cpp
are missing. After fixing the Jamfile accordingly I get my precious try_signal.a
.
After that, linking both with libtorrent-rasterbar.a
and try-signal.a
seems to work.
I'm not entirely sure if I'm doing something wrong or if something is missing for the static build configuration.
It is because you are not using install
. Whether that is intentional or a bug is for @arvidn to say. Here is my proof of concept.
Install to a relative lt-build
directory using these install commands:
build_dir="$(pwd)" && curl -sNLk https://boostorg.jfrog.io/artifactory/main/release/1.76.0/source/boost_1_76_0.tar.gz --create-dirs -o "${build_dir}/lt-build/boost_1_76_0.tar.gz"
tar xf "${build_dir}/lt-build/boost_1_76_0.tar.gz" -C "${build_dir}/lt-build"
pushd "${build_dir}/lt-build/boost_1_76_0/" || exit
"${build_dir}/lt-build/boost_1_76_0/bootstrap.sh"
#
export BOOST_BUILD_PATH="${build_dir}/lt-build/boost_1_76_0"
#
echo "using gcc : $(gcc -dumpversion) : g++-$(g++ -dumpversion) ;" > "$HOME/user-config.jam" # -dumpversion may give return a bad result outside a debian based systems
#
git clone --single-branch --branch RC_2_0 --shallow-submodules --recurse-submodules --depth 1 https://github.com/arvidn/libtorrent "${build_dir}/lt-build/libtorrent"
#
pushd "${build_dir}/lt-build/libtorrent" || exit
And using this build command with NO install
"${build_dir}/lt-build/boost_1_76_0/b2" -j"$(nproc)" optimization=speed crypto=openssl cxxstd=17 variant=release threading=multi link=static boost-link=static
[1] gcc-8
...patience...
...patience...
...patience...
...found 3790 targets...
...updating 185 targets...
compiles...
gcc.archive bin/gcc-8/release/crypto-openssl/cxxstd-17-iso/link-static/threading-multi/libtorrent-rasterbar.a
...updated 185 targets...
If i look for the libtry_signal.a
lib it is not found
server:~/lt-build/libtorrent$ find . -name libtry_signal.a
server:~/lt-build/libtorrent$
Now with the install command install --prefix="${build_dir}/lt-build/libs"
"${build_dir}/lt-build/boost_1_76_0/b2" -j"$(nproc)" optimization=speed crypto=openssl cxxstd=17 variant=release threading=multi link=static boost-link=static install --prefix="${build_dir}/lt-build/libs"
I see this:
[1] gcc-8
...patience...
...patience...
...patience...
...found 4346 targets...
...updating 477 targets...
compiles...
common.copy /home/username/lt-build/libs/lib/libtorrent-rasterbar.a
...updated 477 targets...
If I look for the libtry_signal.a
lib it is found and also install to lt-build/lib
server:~/lt-build/libtorrent$ find . -name libtry_signal.a
./deps/try_signal/bin/gcc-8/release/crypto-openssl/cxxstd-17-iso/link-static/threading-multi/libtry_signal.a
server:~/lt-build/libtorrent$
So use the install
option.
Unless there is a flag I am missing to build it like the python module has build-dependencies
but i don't think there is.
It is because you are not using install. Whether that is intentional or a bug is for @arvidn to say. Here is my proof of concept.
I think it's reasonable to not build the target since nothing depends on it. Once you invoke the install rule, it has a dependency on everything and it will build it.
@3unjee
That didn't stop me from building it manually with the very same b2 command. Unfortunately, that fails because test.cpp and example.cpp are missing. After fixing the Jamfile accordingly I get my precious try_signal.a.
What changes did you have to make to which Jamfile? what test.cpp and example.cpp are missing?
What changes did you have to make to which Jamfile? what test.cpp and example.cpp are missing?
When building try_signal
manually I needed to patch the Jamfile because it was asking for test.cpp and exemple.cpp from the original repository to build. But I'm not supposed to do that anyway.
As for my issue, adding target-os=android
and deploying in the build directory with the install
option did the trick.
The following command seems to solve it for me:
b2 clang-arm -j4 cxxflags="-fPIC -DANDROID" cxxstd=14 variant=release link=static target-os=android install --prefix="$PWD/build"
As a sidenote, I didn't need that target-os
flag when building libtorrent 1.2.x on Android.
Thank you @arvidn @userdocs
would someone mind testing this? https://github.com/arvidn/libtorrent/pull/6234 It's supposed to install the .pc file into libdir/pkgconfig
It worked for me.
build_dir="$(pwd)" && curl -sNLk https://boostorg.jfrog.io/artifactory/main/release/1.76.0/source/boost_1_76_0.tar.gz --create-dirs -o "${build_dir}/lt-build/boost_1_76_0.tar.gz"
tar xf "${build_dir}/lt-build/boost_1_76_0.tar.gz" -C "${build_dir}/lt-build"
pushd "${build_dir}/lt-build/boost_1_76_0/" || exit
"${build_dir}/lt-build/boost_1_76_0/bootstrap.sh"
export BOOST_BUILD_PATH="${build_dir}/lt-build/boost_1_76_0"
echo "using gcc : $(gcc -dumpversion) : g++-$(g++ -dumpversion) ;" > "$HOME/user-config.jam" # -dumpversion may give return a bad result outside a debian based systems
git clone --single-branch --branch pkg-config-path --shallow-submodules --recurse-submodules --depth 1 https://github.com/arvidn/libtorrent "${build_dir}/lt-build/libtorrent"
pushd "${build_dir}/lt-build/libtorrent" || exit
"${build_dir}/lt-build/boost_1_76_0/b2" -j"$(nproc)" optimization=speed crypto=openssl cxxstd=17 variant=release threading=multi link=static boost-link=static install --prefix="${build_dir}/lt-build"
Results
server:~/lt-build/lib$ ls -R
.:
libboost_system.a libtorrent-rasterbar.a pkgconfig
./pkgconfig:
libtorrent-rasterbar.pc
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Greetings, I'm having the following issue when linking against a static libtorrent-rasterbar 2.0.3 for Android x86_64.
The funny thing is that it works perfectly with arm32 and x86. This seems to be 64bit related with
sig::detail / sig::errors
.I'm building libtorrent with the following:
b2 clang-arm -j4 cxxflags="-fPIC -DANDROID" cxxstd=14 variant=release link=static openssl-version=pre1.1
, the compiler isx86_64-linux-android29-clang++
.UPDATE: It seems
try-signal
does not get built with the b2 line just above, thus missing in the static library.Here is the log: