search

Tribler / tribler

Privacy enhanced BitTorrent client with P2P content discovery
https://www.tribler.org
GNU General Public License v3.0
4.84k stars 449 forks source link

Provide TriblerCore builds for ARM Linux #5327

Open hbiyik opened 4 years ago

hbiyik commented 4 years ago

Hello Team

Problem 1: A conveinent way for tribler headless (triblerd) should be provided to deploy the server without QT and UI stuff. Currently run_tribler.py can be run with some ENV variables (CORE_PROCESS etc..) and which is not very conveinent. Another approach is there run_tirbler.headless.py which is nice way but does not support versioned dirs. I should this part needs to be refactored.

Problem2: How to build triblerd for arm variants of Linux. Good news. I already have compiled for ARM! yay, here is how it works:

ARM buildsys for tribler: a debian(wheezy) based image, with arm 32bit soft floating point toolchain (armel triplet) is run under QEMU emulator. All the necessary compenents are built in the emulator (libtorrent, boost, openssl etc.).

This environment gives us a very broad support with >libc 2.13 and arm support over > armv5, practically all 32bit arm socs. (I remember 64bit arms to work with 32bit ISA as well but i need to check).

The downside is, since we are running in an emulator, compilation speed is quite low, but thats ok, we compile once for the dependencies and freezing tribler is not everydays work.

Another downside is, even though soft float arm abi, gives a broad support for SOCs, it runs slower in most of the devices where hard floating point is available (> armv6 basically). This will be fixed with another emulator container with armhf triplet but in first stage i want to work with armel to see the performance and application scope etc..

But to move forward I need a solution for the first problem and decide what to build and deliver.

synctext commented 4 years ago

cool progress. Would it help if we at some point procured a an ARM laptop for our build farm? https://www.pine64.org/pinebook/ Native execution..

hbiyik commented 4 years ago

That device is using an arm64, i checked that aarch64 can run isa from armv5 in 32bit as well.

Together with docker containers should help i think, with containers there should be an abstract approach to have both armel, armhf and aarch64 toolchains on the same device.

hbiyik commented 4 years ago

I did further investigation and i feel kind of silly for trying to build that qemu image for 3 days since docker has already a feature to emulate the ISA of different CPU architectures and pass to kernel on runtime. This is called binfmt support with QEMU, meaning that you can run guest instructions with full host cpu capabilities in emulation mode without emulating a hardware or SOC but only emulating the ISA. So there is no need for an ARM native device also, i think this should be fairly fast.

This is something new to me, may be it was there since for a long time but sometimes you learn the hard way :). Nevertheless problem 1 still stands.

synctext commented 4 years ago

versioned dirs? those are the state directories for .Tribler, isolated between upgrades. Feel free to move that code for Triblerd re-usage purposes. Any dev suggestions?

hbiyik commented 4 years ago

a side update on buildsys: https://github.com/hbiyik/triblerd-buildsys/blob/master/dockerfiles/debian-lenny-armel.dockerfile

This container builds the tribler in arm ISA and can work on x86 host, works like a charm.

You can even integrate the foreign CPU python to your IDE (ie eclipse), and run the debugger inside the container which is awesome, or you can deploy the python files of tribler in the container and distribute container. I liked this.

Also worth to mention that container is based on debian lenny, which was released in 8 years ago :) Glibc dependency is 2.7, currenlty glibc is on 2.31 !

hbiyik commented 4 years ago

The buidlsys is ready, and finally i can run tirbler in linux arm, Yay!

With this buildsys, we can run tribler in a linux basically from last 15-20 years! (except uclibc variants)

You can find the buildsys in:

https://github.com/hbiyik/triblerd-buildsys

Also the binaries generated:

https://github.com/hbiyik/triblerd-buildsys/releases/tag/7.5-master

Below is the toolchain information, and version summary used in the compilation progress. https://github.com/hbiyik/triblerd-buildsys/blob/master/toolchain.json

{
  "images": {
    "linux-i386": {
      "base": "debian/eol:lenny-slim@sha256:a2230f615ba41081b6727b32234275064e4f53de77dc94765fcaa24489d6f930",
      "openssl-platform": "linux-x86",
      "qemu-platform": "i386",
      "toolchain": {
        "gcc": "4.3.2",
        "glibc": "2.7"
      }
    },
    "linux-amd64": {
      "base": "debian/eol:lenny-slim@sha256:4211c70bd47b74942957b208e9f54cb52eaa15b0b48c3d76f3f9b502929d1d01",
      "openssl-platform": "linux-x86_64",
      "qemu-platform": "x86_64",
      "toolchain": {
        "gcc": "4.3.2",
        "glibc": "2.7"
      }
    },
    "linux-armel": {
      "base": "debian/eol:lenny-slim@sha256:859461d1c5e8e39244a803fef8cad2b840cd9198542d7edc0bada781f5794b1a",
      "openssl-platform": "linux-generic32",
      "qemu-platform": "arm",
      "toolchain": {
        "gcc": "4.3.2",
        "glibc": "2.7"
      }
    },
    "linux-armhf": {
      "base": "debian/eol:wheezy-slim@sha256:54a2fa7015ca274803d8e64f622a375057efa89d3595bb9148d62aa657b3b3c8",
      "openssl-platform": "linux-armv4",
      "qemu-platform": "arm",
      "toolchain": {
        "gcc": "4.6.3",
        "glibc": "2.13"
      }
    },
    "linux-aarch64": {
      "base": "debian/eol:jessie-slim@sha256:600ca584836c903364017aea8f4ca4af335f04c949a7dd78801477cf3cb6cdc5",
      "openssl-platform": "linux-aarch64",
      "qemu-platform": "aarch64",
      "toolchain": {
        "gcc": "4.9.2",
        "glibc": "2.19"
      }
    }
  },
  "deps": {
    "perl": {
      "version": "5.30.2",
      "dir": "sources/perl-5.30.2"
    },
    "openssl": {
      "version": "1.1.1",
      "dir": "sources/openssl-1.1.1g"
    },
    "Python": {
      "version": "3.7.7",
      "dir": "sources/Python-3.7.7"
    },
    "boost": {
      "version": "1_53_0",
      "dir": "sources/boost_1_53_0"
    },
    "libtorrent_rasterbar": {
      "version": "1.1.5",
      "dir": "sources/libtorrent-rasterbar-1.1.5"
    },
    "libsodium": {
      "version": "1.0.18",
      "dir": "sources/libsodium-1.0.18"
    },
    "cryptography": {
      "version": "2.9.2",
      "dir": "sources/cryptography-2.9.2"
    },
    "sqlite_autoconf": {
      "version": "3320100",
      "dir": "sources/sqlite-autoconf-3320100"
    }
  },
  "interpreter": "python3.7"
}
hbiyik commented 4 years ago

@devos50

How is the current buildsystem for windows binaries? Is there any docker image, i think a linux container over wine + python3.7 is possible..

devos50 commented 4 years ago

@hbiyik we are using PyInstaller to bundle the Python executable/runtime, and create an installer with NSIS.

hbiyik commented 4 years ago

I mean is there any compilation environment like docker like there is for linux? Or precompiled binaries are used for openssl, boost, libtorrent, libsodium, or some windows machine?

devos50 commented 4 years ago

No, we use pre-compiled libraries. We have compilation instructions, though.

hbiyik commented 4 years ago

Ok, i see, I have a funny experiment to compile all deps in a linux container, and freezing with wine, so that each platform release will have the same c code, lets see how that will work

devos50 commented 4 years ago

We can test the outcome of your experiment on our Windows machines if you wish :)

hbiyik commented 4 years ago

Would be great, I have also made some experiments on android as well, successfully builds and it kind of works, but i expect bunch of errors on runtime when all the test suite is run mainly from psutil and some other libs.

Here is a snapshot from android logcat and buildozer:

image

hbiyik commented 4 years ago

Very good news. I have managed to run unittests in android as well.

6 of 659 tests failed. Mainly due to psutil, and some netowrk socket related things, shouldnt be so hard to wrap around it.

I think android port is almost ready, there is not much of workload here.

Tribler Core: 659 Tests, 6 Failed Anydex : 267 Tests. 12 Failed (due to bitcoinlib, those tests dont work on pc also so no issues on android) IPv8: There are some import isseues there, could not run yet, but dont expect a biggie Total tested: 926

Here is the overall output and traceback of all tests:

https://pastebin.com/raw/5cErn8DJ

hbiyik commented 4 years ago

@synctext @ichorid @devos50 @egbertbouman I think i am finished with the buildsystem, and can finally produce binaries for arm/x86 linux and arm/x86 android.

Could you take your time and read the readme file in https://github.com/hbiyik/triblerd-buildsys

which details why, how and what.

PS: if you really want to run in android, there is a small fix to be done first as below: https://github.com/hbiyik/tribler/commit/1d4ed4ad260ee78cd40c39c3e0bcd838b338d3d9

synctext commented 4 years ago

wow. Very elaborate documentation. Worked with cross compilers, but this is not easy to grasp. How difficult is a download/upload performance table to make for a few boards?

hbiyik commented 4 years ago

I tried to put a lot of detail in the documentation cos i know it is quite a tricky topic and some of the things are based on historical event chains (like glibc dependencies), in case a student may take over, he can have a holy bible to consult instead of praying :)

but never the less all this complexity is reduced to some basic commands in the case of usage.

I tested with an armv8 cpu and the speed can easily get over 1MByte/s with 1 hop encrypted tunnels. What would be great is to have a unit testing for those benchmarks so the results would be comperable.

I have a pi-1, which is great to test the behaviour in absolute lowest device, 2 android phones with decent 8core cpus.

synctext commented 4 years ago

The complexity is quite something, your expertise is vital for maintenance. Kodi is a big new community, millions of users. Some would ask us for their favourite new feature.

1 MByte/sec with 1 tunnel hop, impressive results. We have numerous loose Android devices (10+), but no testing infrastructure. Unclear to me how far away you are from .APK builds. Like for high-end octa-core (4x2.4 GHz Cortex-A73 & 4x1.8 GHz Cortex-A53).

hbiyik commented 4 years ago

I can always support in such cases, no issues.

Anyone with a linux x64 pc can already build the apk with this already. For testing, before bundling the core into apk, better to patch this line from localhost to 0.0.0.0

https://github.com/Tribler/tribler/blob/51dc33b09846fdd533df3ccd2fa7c25b397629bb/src/tribler-core/tribler_core/restapi/rest_manager.py#L100

and after running the apk, control the core over ip.of.the.device:8085 from a desktop pc and do fancy testing.

See below how core is initialized in android, it is basically triblerr_headless.py

https://github.com/hbiyik/triblerd-buildsys/blob/master/buildozer/main.py

ichorid commented 4 years ago

Now the question is: how do we integrate this into our devops/release cycle?...

devos50 commented 4 years ago

@hbiyik impressive work! Sorry for the delayed response; we have been/are quite busy with scientific output the past few weeks so engineering efforts were a bit low. This line of work therefore did not get the attention that it deserves. From experience, I know that working with P4A/cross-compilation is not easy and we always struggled to get a proper build process for Android up and running. I will check out the readme and try to get it up and running later today.

Integration in our Jenkins pipelines should be the next step, and most likely requires further discussion/granting you some permissions to our infrastructure. We used to have a few Android phones hooked up to one of our computers, but they died recently. Can we run the unit tests in an Android simulator first? After some infrastructure improvements, I hope we can run these tests on a real device. 👍

If you wish, I can invite you to our Slack channel so we can have a more real-time discussion about this (please send me an email if you want this invitation).

Also, note that one of our top master students is working on a Kotlin port of Tribler in parallel. You might want to check out that app too.

devos50 commented 4 years ago

@hbiyik Also, I suggest moving the repository to the Tribler organization, if you are ok with that 👍

hbiyik commented 4 years ago

@devos50 Ok i think i need access rights for this transfer thing, or you can fork it if you want.

I know that working with P4A/cross-compilation is not easy and we always struggled to get a proper build process for Android up and running

I know what you mean i saw a lot of foot prints of @brussee in p4a repo, thanks to him he has already done the heavy lifting. But things have changed and now we have an excellent tool like docker, where we can freeze the environment and move it around freely, and even emulate remote foreign cpu instructuion sets. Now things are much way easier.

Can we run the unit tests in an Android simulator first?

If theere is adb access then you should yet i tested on real devices and it definetely works, after some fixing obvious stuff in tribler source. (mentioned in above comments)

Can we run the unit tests in an Android simulator first?

Would love to, add me at huseyinbiyik at hotmail.

@ichorid Firstly I would like to point out another use case for this buildsystem and how tribler application is distribute / deployed.

The generated core binaries are single files, with a filesize around 25 ~ 35 MB.

It is possible to distribute the QT gui only, and the gui itself may check the binary repo and download the latest core binary and execute it as a service. This way whenever there is a new version of core, the user wont need to update the Tribler but tribler will automatically do it, and this will significantly reduce the release cycle time, and help to overcome user resistance on system upgrades.

Below is a sample code for detecting which binary/toolchain to use in linux in runtime, a very basic elf parser: https://github.com/hbiyik/repository.biyik/blob/master/plugin.program.tribler/tribler/core.py#L23

ichorid commented 4 years ago

It is possible to distribute the QT gui only, and the gui itself may check the binary repo and download the latest core binary and execute it as a service. This way whenever there is a new version of core, the user wont need to update the Tribler but tribler will automatically do it, and this will significantly reduce the release cycle time, and help to overcome user resistance on system upgrades.

That is a very interesting idea, indeed.

Currently, there is a variety of methods of application deployment on Linux platforms (e.g. Snap vs Docker vs package manager vs pip vs whatever...) We should carefully select the most efficient one or two and support these. The problem with this stuff is that supporting a variety of methods will spread the team's effort even thinner than it is now, and can confuse users.

We would prefer not to touch the existing build system until fresh blood arrives in September, though.

hbiyik commented 4 years ago

My motivation was to have a uniform way of having a deployment mechanism of the daemon for kodi users, and came up with this, I will surely use this method because it really helps to reduce all hose complexity in case of a new tribler release, may be i can use this as a trial in kodi user base and collect the feedback. Since http serving over github is totally free :)

hbiyik commented 4 years ago

just added windows to toolchains.

now 2 more targets are available

windows-i386 windows-amd64 linux-arm6hf

only thing that this buildsystem cant package to from the same linux distro is now OSX. I would like play dead when apple stuff is in place :)