Tribler / tribler

Privacy enhanced BitTorrent client with P2P content discovery
https://www.tribler.org
GNU General Public License v3.0
4.86k stars 451 forks source link

Rename *_all.deb in release to *_x64.deb #8171

Closed Dmole closed 1 month ago

Dmole commented 2 months ago
wget https://github.com/Tribler/tribler/releases/download/v7.14.0/tribler_7.14.0_all.deb

apt install ./tribler_7.14.0_all.deb

file /usr/share/tribler/tribler
/usr/share/tribler/tribler: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3dfe013b3027047470d1aac10a3504baf6969725, for GNU/Linux 2.6.32, stripped

uname -m
aarch64
qstokkink commented 2 months ago

Thanks for the suggestion. Just in time for the 8.0 release 👍

qstokkink commented 2 months ago

Alright, I looked into what generates this name. It seems that our build name is left completely to dpkg-buildpackage:

https://github.com/Tribler/tribler/blob/8c457b9fda6373000efbbf633dd5f49602974a12/build/debian/makedist_debian.sh#L29

If I interpret the manpage correctly:

The only way I see the output name including the arch, is if we switch to --build=any. I'm not entirely sure of the implications but - at first glance - it seems equivalent to what we build now.


Nope, that breaks everything. Perhaps there is some other command to specify the name?

qstokkink commented 2 months ago

I burned a few hours on this. Long story short, for the upcoming release I think we should just rename the output manually.

This ties into #5345 and #5327. If we want to do this the "right" way (like the Debian docs recommend, giving us proper file names), we should either (a) ship the source in the .deb and build it on whatever machine it ends up in, for a true "_all" distribution, or (b) provide binary builds for different platforms. Currently, as OP shows, we bundle one specific binary and pretend that it is applicable for "all": this is bad. That said, this will take quite a bit of effort to fix and I don't think right now is the time to do so. Some time after the 8.0 release would be better.

I will try a bit more to get a "proper" amd64 binary build. So far, I'm still looking for the magic config though.

qstokkink commented 2 months ago

For the simple output renaming, I have the following ugly-but-automated solution:

diff --git a/build/debian/tribler/debian/rules b/build/debian/tribler/debian/rules
index d760a8242..70adbe290 100755
--- a/build/debian/tribler/debian/rules
+++ b/build/debian/tribler/debian/rules
@@ -13,6 +13,29 @@
 # package maintainers to append LDFLAGS
 #export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed

+DEB_HOST_ARCH ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)
+DEB_SOURCE_PACKAGE := $(strip $(shell egrep '^Source: ' debian/control | cut -f 2 -d ':'))
+DEB_VERSION := $(shell dpkg-parsechangelog | egrep '^Version:' | cut -f 2 -d ' ')
+
+dh binary:
+       dh_testroot
+       dh_prep
+       dh_auto_install --destdir=debian/tribler/
+       dh_install
+       dh_installdocs
+       dh_installchangelogs
+       dh_installmenu
+       dh_icons
+       dh_perl
+       dh_link
+       dh_strip_nondeterminism
+       dh_compress
+       dh_fixperms
+       dh_missing
+       dh_installdeb
+       dh_gencontrol -- "-n${DEB_SOURCE_PACKAGE}_${DEB_VERSION}_${DEB_HOST_ARCH}.deb"
+       dh_md5sums
+       dh_builddeb --filename=${DEB_SOURCE_PACKAGE}_${DEB_VERSION}_${DEB_HOST_ARCH}.deb

 %:
        dh $@

Still looking for something nicer. I feel like there should be a better solution.

qstokkink commented 2 months ago

I don't think it's worth delaying 8.0.0 for this, I'll reschedule for 8.1.0 and just do a manual rename.

qstokkink commented 1 month ago

Documenting my random searches. I came across two promising ways to handle compilation on other architectures for use with GitHub Actions.

qstokkink commented 1 month ago

I think I have a successful aarch64 run with run-on-arch-action:

https://github.com/qstokkink/tribler/actions/runs/11255571313

tribler: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=fd10de9ef52828387bf3bc659e52b4587c9d2c42, for GNU/Linux 3.7.0, stripped

However, I have no way to test this. @Dmole are you willing to test the build?

Dmole commented 1 month ago
uname -m
aarch64

lsb_release -d
Description:    Ubuntu 22.04.5 LTS

/usr/share/tribler/tribler -s
/usr/share/tribler/tribler: error while loading shared libraries: libcrypt-06cd74a6.so.2: cannot open shared object file: No such file or directory

That should be just "libcrypt.so" (a static build may be a security issue, as would Flatpack etc).

qemu or cloud providers (like amazon) can be used for aarch64 testing.

qstokkink commented 1 month ago

Thanks for testing 👍 I had hoped that getting the build running under a different arch was enough. However, it seems I'll have to get deep into how cx_Freeze pulls in shared objects.

Dmole commented 1 month ago

Why use cx_Freeze at all; Why not the good old platform independent "python tribler.py" on Linux/MacOS/BSD?

Dmole commented 1 month ago

Something (cx_Freeze? dpkg-buildpackage?) breaks reproducible builds :/

qstokkink commented 1 month ago

I do see the appeal of zero-effort releases by just shipping Python, but there are some issues with Python's import system that can cause a nightmare when trying to debug issues (let alone people making custom patches). For example, I recall when our networking library imported half of its files from a system-wide install and the other half from the local directory. Especially when limited to a reported traceback, it is nice to have a stable version and isolation.

For more experienced people, like yourself I think, the added value of a build is probably a lot less.

I think dpkg-buildpackage adds timestamps in some files. So, the build should never be fully reproducible.

qstokkink commented 1 month ago

I integrated a test run into the builder action and I can now reproduce the missing libcrypt-06cd74a6.so.2. I'll try to play around with this and see if I can get it to work.

- uses: uraimo/run-on-arch-action@v2
  name: Run commands
  id: runcmd
  with:
    arch: aarch64
    distro: ubuntu22.04
    githubToken: ${{ github.token }}
    dockerRunArgs: |
      --volume "${PWD}:/tribler"
    env: |
      GITHUB_TAG: '8.0.1'
    shell: /bin/sh
    install: |
      apt-get update -q -y
      apt-get install -q -y --allow-downgrades alien cpio=2.13+dfsg-7 devscripts fakeroot gir1.2-gtk-4.0 libgirepository1.0-dev rpm python3-pip libcairo2-dev
    run: |
      cd /tribler
      python3 -m pip install --upgrade -r build/requirements.txt
      python3 -m pip install meson ninja
      export PATH="/usr/local/bin:$PATH"
      ./build/debian/makedist_debian.sh

      cd build/debian
      apt-get install -y --fix-broken ./tribler_8.0.1_all.deb
      timeout 10s tribler -s || true
      cat /tmp/*tribler*.log

Getting back to the original issue, my goal was to see what the output name would be. The answer is that this also produces a tribler_8.0.1_all.deb. So, to get the proper name for the .deb, the most sane thing to do seems to be to simply rename tribler_8.0.1_all.deb to tribler_8.0.1_amd64.deb after building the step here:

https://github.com/Tribler/tribler/blob/c296e08f33058266b8e092edfaffa18c594aa665/.github/workflows/build.yml#L100