master255 commented 3 years ago

As I see it, the 32-bit version has a minimum version of 21. It needs to be version 19.

master255 commented 3 years ago

@aldenml I'm talking about this bug https://github.com/arvidn/libtorrent/issues/4338

aldenml commented 3 years ago

I'm open to exploring the creation of a build with API-19, maybe with a different artifact. Out of curiosity, is your user base with API-19 a significant amount?

master255 commented 3 years ago

@aldenml No. There are few users with Android 4.4, but they are about 4%. And I have a Galaxy S3 test phone (which has outstanding specs: support wifi 5G, has a 4x core processor and NFC technology) that runs on Android 4.4.

Other than that, one line of code is not a reason to raise the minimum api version. It's not professional.

master255 commented 3 years ago

@aldenml Ha ha. We still have these phones on sale. Just found it...didn't know about it. https://www.e-katalog.ru/SAMSUNG-GALAXY-S3-16GB.htm

master255 commented 3 years ago

@aldenml And jlibtorrent-1.2.10.0 supported Android 4.4. I just checked it. But jlibtorrent-1.2.12.0 doesn't. The problem is in the compilation scripts.

master255 commented 3 years ago

@aldenml I found the problem! The thing is that the toolchain for the 21 api is being used. And we need to use toolchain for 19 api. And then these commands will work:

hacks to remove when API >= 21

<cxxflags>"-D\"epoll_create1(x)=-1; errno=ENOSYS\"
<cxxflags>"-D\"posix_fadvise(x1,x2,x3,x4)\""

Fix this. Tell me how to do a release version with your script and we'll move to torrents 2.0.

aldenml commented 3 years ago

OK, doing some experiments with API 19. Btw, keep in mind that there are some non-trivial API differences between libtorrent4j and jlibtorrent since we track different libtorrent branches.

I will let you know the results.

master255 commented 3 years ago

@aldenml I don't use much frost wire code. Almost everywhere I have a direct call to the library. I am waiting for your results. Maybe this will help you: https://github.com/frostwire/frostwire-jlibtorrent/issues/250#issuecomment-766350633

aldenml commented 3 years ago

@master255 this is what I have so far:

With some tricks, I can build the arm and x86 for android api-19. See the branch https://github.com/aldenml/libtorrent4j/tree/android-19, you can download the binaries artifacts here https://github.com/aldenml/libtorrent4j/runs/1758351202.
The problem is more than just the lack of posix_fadvise. The new libtorrent >= 2.x uses the posix call fseeko, and with "large files" support this only becomes available with API >= 24. Still, it could be present at runtime with the device's own bionic. You need to test multiple devices (with API 19) to see how likely the success rate is.
As pure extra information, Android KitKat (and below) is full of corner cases when it comes to large files (i.e. > 4GB). See: https://android.googlesource.com/platform/ndk/+/master/docs/user/common_problems.md#using-with-early-api-levels, https://android-developers.googleblog.com/2017/09/introducing-android-native-development.html and https://android.googlesource.com/platform/bionic/+/master/docs/32-bit-abi.md.

I tried to run emulators with API-19, but for some reason, they don't work anymore on my PC. If you are willing to try with the binaries, I can take a look at the log if you have some errors.

I mentioned the difference between libtorrent4j and jlibtorrent (api wise) because they are not 100% compatible (even to the swig layer). I recommend you to try changing the dependency to libtorrent4j in your build script and see if it even compiles, then see if you are comfortable with the number of necessary changes.

Hope this helps.

gubatron commented 3 years ago

I think we'll be dropping Kit Kat 4.4 (android api 19) support as of our next build. Stats for 2002 put Kit Kat at 4.4% of the market share (statista) and 4.2% for FrostWire's userbase. At these points those users are having more problems than most and leaving negative reviews.

gubatron commented 3 years ago

I've been battling this issue when building 32-bit for arm at link time:

requires unsupported dynamic reloc R_ARM_REL32; recompile with -fPIC

and this error for x86 (32bit) also at link time

relocation R_386_GOTOFF against preemptible symbol _ZN5boost4asio6detail10call_stackINS1_14thread_contextENS1_16thread_info_baseEE4top_E cannot be used when making a shared object

My android-arm-config.jam has as the first option -fPIC, and the 64-bit versions for android build and link just fine.

I'm building on a 64-bit Linux machine, I wonder if this has anything to do with that.

At this point I'm willing to run a 32bit virtual machine, or just skip altogether local builds and port the build scripts to github actions. Perhaps it's faster trying the 32bit vm.

import os ;

ANDROID_TOOLCHAIN = [ os.environ ANDROID_TOOLCHAIN ] ;
ANDROID_API = [ os.environ android_api ] ;

using clang-linux : arm : $(ANDROID_TOOLCHAIN)/bin/armv7a-linux-androideabi$(ANDROID_API)-clang++
    <cxxflags>-fPIC
    <cxxflags>-march=armv7-a
    <cxxflags>-mfpu=neon
    <cxxflags>-std=c++14
    <cxxflags>-DANDROID
    <cxxflags>-D__STDC_FORMAT_MACROS
    <cxxflags>-D_FILE_OFFSET_BITS=64
    <cxxflags>-frtti
    <cxxflags>-fno-strict-aliasing
    <cxxflags>-fvisibility=hidden
    <linkflags>-static-libstdc++
    <linkflags>-fuse-ld=bfd
    # debug information
    <cxxflags>-g
    <cxxflags>-gdwarf-4
    <cxxflags>-fdebug-macro
    <cxxflags>-ggdb
    ;

gubatron commented 3 years ago

Thanks @master255 I was missing the ":" at the end of using clang-linux : arm : $(ANDROID_TOOLCHAIN)/bin/armv7a-linux-androideabi$(ANDROID_API)-clang++ like you mentioned on the jlibtorrent issue tracker.

This made the arm 32-bit finally built!

The windows build is still broken (even after adding the also missing ":" to that .jam file)

master255 commented 3 years ago

@aldenml I tested it. And this is what I got:

The 1.2 version of the library works well on my Galaxy S3. Even with files over 4 gigabytes.
Your script does not compile arm 32 version of the library. The error is in the attached file.
I still don't understand how to get a release version of your library. Maybe I need to use some similar script? https://github.com/frostwire/frostwire-jlibtorrent/blob/master/swig/package-remote-build.sh
In order to test your library I need to modify it for streaming (and other features) and compile it myself. Your usual version of the library will not work for me exactly. How can I do that?

master255 commented 3 years ago

@aldenml

Okay. I repeated the fork and the compilation was successful.

Answer the other questions.

aldenml commented 3 years ago

@master255

1- Good to know you have a device with no issues 2- OK 3- I haven't figure out a way to download the binaries with a script, but you can do it manually if you build them using GitHub Actions.

4- How do you it right now using jlibtorrent? If you are using some sort of script for the local build, I imagine I can take a look and adapt it for libtorrent4j.

master255 commented 3 years ago

@aldenml Perhaps we need to develop scripts to compile on the local computer? So as not to download 300 megabytes and not to wait so long to compile after changing one line of code. As it is done in https://github.com/frostwire/frostwire-jlibtorrent

Yes. I am currently compiling Jlibtorrent on my local computer. The resulting jar files are 2.5 Megabytes in size. This is as quick and easy as possible for me. I have a very powerful computer and I can control the compilation process. This post contains my swig directory with which I was able to compile the latest version 1.2.12 of libtorrent.

I am using Windows 10 Compact and as a Windows component Ubuntu 20.04 LTS.

I tried to change the libtorrent branch to 2.0, but errors appeared. So far I don't know how to fix them. If you can help with these scripts, it will help us to switch to 2.0 version.

aldenml commented 3 years ago

OK, I will take a look at the scripts and report back

gubatron commented 3 years ago

Beware of local compilation when distributing binaries.

Our last build for Linux for instance was built in my shiny new Ubuntu kernel, and now I have users complaining that they don't have the latest libc :p

Make sure to build on older kernels whatever you build to maximize binary compatibility. #LessonsLearned.

gubatron commented 3 years ago

(And I still cannot build jlibtorrent for 32bit x86 android, can't link the damn thing)

master255 commented 3 years ago

@gubatron

(And I still cannot build jlibtorrent for 32bit x86 android, can't link the damn thing)

What's your error?

aldenml commented 3 years ago

Hi @master255, copying from jlibtorrent, I created a script for the local build of arm 32bits. I don't want to go further until you are able to test at least with that architecture.

You need to checkout the branch android-19 (https://github.com/aldenml/libtorrent4j/tree/android-19). Inside the folder swig/scripts-b2 (https://github.com/aldenml/libtorrent4j/tree/android-19/swig/scripts-b2) you will find the very familiar script build-android-arm.sh. Give it a try and let me know.

master255 commented 3 years ago

@aldenml ok

master255 commented 3 years ago

@aldenml I tested with different settings. These errors occur all the time. Can you help fix them? errors.txt

You forgot these lines:

hacks to remove when API >= 21

<cxxflags>"-D\"epoll_create1(x)=-1; errno=ENOSYS\"
<cxxflags>"-D\"posix_fadvise(x1,x2,x3,x4)\""

aldenml commented 3 years ago

I see, go to the line https://github.com/aldenml/libtorrent4j/blob/android-19/swig/scripts-b2/build-android-arm.sh#L20

# NOTE: for Linux, remove "", for macOS, set: sed -i "" <expr>
sed -i "" 's/24/19/g' ${ANDROID_TOOLCHAIN}/sysroot/usr/include/ifaddrs.h
sed -i "" 's/24/19/g' ${ANDROID_TOOLCHAIN}/sysroot/usr/include/stdio.h
sed -i "" 's/28/19/g' ${ANDROID_TOOLCHAIN}/sysroot/usr/include/sys/random.h
sed -i "" 's/POSIX_FADV_RANDOM 1/POSIX_FADV_RANDOM_NOOP 0/g' ${ANDROID_TOOLCHAIN}/sysroot/usr/include/linux/fadvise.h
sed -i "" 's/RANLIB = ranlib/RANLIB = "${ANDROID_TOOLCHAIN}\/bin\/arm-linux-androideabi-ranlib"/g' ${BOOST_ROOT}/tools/build/src/tools/gcc.jam

remove the """ from each sed line. With those changes there is no need to mock epoll_create1 and posix_fadvise.

master255 commented 3 years ago

@aldenml Yes. It's finally working! But at the end there was some problem with the compilation gradle. success.txt

aldenml commented 3 years ago

Can you type gradle tasks inside the folder libtorrent4j1 to see what happens?

master255 commented 3 years ago

@aldenml

aldenml commented 3 years ago

Try ./gradlew tasks please. I'm trying to figure out if it's a gradle incompatibility on your system, since I'm using the Kotlin DSL.

master255 commented 3 years ago

@aldenml After the command ./gradlew tasks The installation of gradle is started. But compiling the library still gives the same errors. log.txt

master255 commented 3 years ago

again

aldenml commented 3 years ago

I think I know what it is. Your system gradle does not support the Kotlin DSL, you need to use the wrapper (./gradlew). I can modify the script, but it's just a simple change:

In these lines https://github.com/aldenml/libtorrent4j/blob/android-19/swig/scripts-b2/build-utils.shinc#L329

gradle clean
gradle build
# android Arm
gradle nativeAndroidArmJar

put

./gradlew clean
./gradlew build
# android Arm
./gradlew nativeAndroidArmJar

master255 commented 3 years ago

@aldenml Yes. You're right. Now everything compiled. But why is the library so big? It was 2 megabytes bigger.

aldenml commented 3 years ago

@master255 good to know you are finally able to compile.

The first reason it is bigger is because libtorrent4j tracks the libtorrent's master branch, which includes WebTorrent. For that to work, I compile openssl with almost all the features enabled. The second reason is that I'm using -O3 instead of -Os.

master255 commented 3 years ago

@aldenml What is this "I'm using -O3 instead of -Os"? Explain in simple words?

I compile openssl with almost all the features enabled

What are the advantages?

aldenml commented 3 years ago

When I compile all the c++ source code, I pass the flag -O3 to the compiler, which optimizes for speed at the cost of bigger binaries. jlibtorrent uses -Os instead, which optimizes for binary size instead.

The reason I enable almost all features in openssl, is because WebTorrent uses a lot of protocols and algorithms while negotiating with web peers. For me to discover which one I can disable, will take me a long long time.

master255 commented 3 years ago

@aldenml Thank you so much! +1000 points in karma 👍

I will test this library on my project later. If any questions arise that I cannot solve, I will write to you.

gubatron commented 3 years ago

my error, only for android x86 32bit relocation R_386_GOTOFF against preemptible symbol

/home/gubatron/src/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/i686-linux-android/4.9.x/../../../../i686-linux-android/bin/ld: error: bin/release/android/x86/libtorrent.a(gzip.o): relocation R_386_GOTOFF against preemptible symbol _ZN5boost6system6detail10cat_holderIvE24system_category_instanceE cannot be used when making a shared object
/home/gubatron/src/android-ndk-r21d/toolchains/llvm/prebuilt/linux-x86_64/bin/../lib/gcc/i686-linux-android/4.9.x/../../../../i686-linux-android/bin/ld: warning: shared library text segment is not shareable

thinking of building on older linux

aldenml commented 3 years ago

@gubatron, are you sure that when you compile the boost_system library, isn't the linux gcc in the path taking precedence over the one in the android toolchain?

master255 commented 3 years ago

@aldenml Okay. I was able to add my changes and compile it. Even runswig worked successfully. While I'm redoing the whole application, I need compile scripts for all platforms. Can you help with these scripts? Android only

master255 commented 3 years ago

@aldenml It is very strange that the data types are different here. That's why I'm having problems.

I found a problem. You need to change the type of the variable. Is this possible?

std::vector<int> get_file_priorities_ex() const
{
    std::vector<libtorrent::download_priority_t> v = $self->get_file_priorities();
    std::vector<int> r(v.size());
    for (std::size_t i = 0; i < v.size(); i++)
        r[i] = int(static_cast<std::uint8_t>(v[i]));
    return r;
}

and:

public Priority[] filePriorities() {
        int_vector v = h.get_file_priorities_ex();
        int size = v.size();
        Priority[] arr = new Priority[size];
        for (int i = 0; i < size; i++) {
            arr[i] = Priority.fromSwig(v.get(i));
        }
        return arr;
    }

in TorrentHandle.java

aldenml commented 3 years ago

@master255 good catch, I will fix that and merge back to theandroid-19 branch. As for the scripts, yes, I can help, but I rather wait until you confirm arm is working in a real device.

master255 commented 3 years ago

@aldenml I finished fixing the application. There are no errors. But here is an error when starting the application:

2-04 03:58:38.674 24242-24242 E/dalvikvm: dlopen("/data/app-lib/com.media.library-15/libtorrent4j.so") failed: dlopen failed: cannot locate symbol "fseeko64" referenced by "libtorrent4j.so"... 02-04 03:58:38.679 24242-24242 W/dalvikvm: Exception Ljava/lang/LinkageError; thrown while initializing Lorg/libtorrent4j/swig/libtorrent_jni; 02-04 03:58:38.679 24242-24242 W/dalvikvm: Exception Ljava/lang/LinkageError; thrown while initializing Lorg/libtorrent4j/swig/settings_pack; 02-04 03:58:38.679 24242-24242 D/AndroidRuntime: Shutting down VM 02-04 03:58:38.684 24242-24242 W/dalvikvm: threadid=1: thread exiting with uncaught exception (group=0x41e95c08) 02-04 03:58:39.204 24242-24287 V/FA: Connection attempt already in progress 02-04 03:58:39.299 24242-24242 E/AndroidRuntime: FATAL EXCEPTION: main Process: com.media.library, PID: 24242 java.lang.LinkageError: Look for your architecture binary instructions at: https://github.com/aldenml/libtorrent4j at org.libtorrent4j.swig.libtorrent_jni.(libtorrent_jni.java:28) at org.libtorrent4j.swig.settings_pack.(settings_pack.java:475)

it's real galaxy s3

aldenml commented 3 years ago

That was what I was afraid of from the very beginning, the lack of fseeko64 at runtime. Now I see two options: 1- You drop "temporarily" support for API-19, I help you with the scripts and later I see how I can avoid that call in libtorrent (might never happen) 2- You don't integrate with libtorrent4j until I figure out a way to modify libtorrent (with no ETA).

master255 commented 3 years ago

@aldenml Bad news. I tried turning on the player on my Sony Bravia Android TV 8 32-bit and got an error: 2021-02-04 04:24:51.273 10253-10342 D/222222: FILE_ERROR 2021-02-04 04:24:51.273 10253-10342 D/222222: Avatar.2009.Extended.UHD.Re-Grade.4000nit.2160p.HEVC.HDR.IVA(RUS.UKR.ENG).ExKinoRay.mkv file_seek (/storage/emulated/0/MediaLibrary/Avatar.2009 Extended.UHD.Re-Grade.4000nit.2160p.HEVC.HDR.IVA(RUS.UKR.ENG).ExKinoRay.mkv-ml) error: Value too large for defined data type

If it does not work on TV, no one will use it. All Android TVs have 32 bits only.

aldenml commented 3 years ago

Sounds to me like the file is too big and you are facing those documented posix related large files issues. Don't you want to try compiling with API-21?

master255 commented 3 years ago

@aldenml Okay, I will try to compile with 21. With version 1.2 this file downloads fine.

master255 commented 3 years ago

@aldenml how to do this?

aldenml commented 3 years ago

I haven't tested yet, but changing this line https://github.com/aldenml/libtorrent4j/blob/android-19/swig/scripts-b2/build-android-arm.sh#L11 to 21 should be enough. Remember to fix the sed lines.

master255 commented 3 years ago

@aldenml Exactly the same: 2021-02-04 04:51:18.755 11133-11225 D/222222: FILE_ERROR 2021-02-04 04:51:18.755 11133-11225 D/222222: Avatar.2009.Extended.UHD.Re-Grade.4000nit.2160p.HEVC.HDR.IVA(RUS.UKR.ENG).ExKinoRay.mkv file_seek (/storage/emulated/0/MediaLibrary/Avatar.2009.Extended.UHD.Re-Grade.4000nit.2160p.HEVC.HDR.IVA(RUS.UKR.ENG).ExKinoRay.mkv-ml) error: Value too large for defined data type 2021-02-04 04:51:18.755 11133-11225 D/222222: TORRENT_ERROR

Libtorrent 2.0 does not work on Android TV. We should tell Arvin about it.

aldenml commented 3 years ago

well, this is very odd then, will try to replicate it on my end

aldenml / libtorrent4j

What about the minimum version of Android? #155

hacks to remove when API >= 21

hacks to remove when API >= 21