search

Azathothas / Toolpacks

Official (pkgforge-edge) Repo 📦📀 & The Largest Collection of Pre-Compiled (+ UPXed) Linux Static Binaries (incl. Build Scripts) & Package Manager (rust) :: https://github.com/pkgforge/soar
https://bin.pkgforge.dev/
MIT License
154 stars 16 forks source link

[PKG REQUEST] SimpleX Chat CLI for x86_64 #34

Open AlbydST opened 2 months ago

AlbydST commented 2 months ago

Repository

https://github.com/simplex-chat/simplex-chat

Description

The first messaging network operating without user identifiers of any kind (See also the CLI version).

Binary Releases?

Yes, available here.

Needs to be compiled?

Yes, releases are dynamic.

Compilation Instructions:

Can be compiled statically as seen here.

Analysis of official binary:

# file simplex-chat-ubuntu-22_04-x86-64
simplex-chat-ubuntu-22_04-x86-64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=9a8b87e6860fd260c2612e14bc71b2e09a20, not stripped
# ldd simplex-chat-ubuntu-22_04-x86-64`:
        /lib64/ld-linux-x86-64.so.2 (0x7f8a3869f000)
    libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f8a3869f000)
    libz.so.1 => /lib/libz.so.1 (0x7f8a38689000)
    libcrypto.so.3 => /lib/libcrypto.so.3 (0x7f8a38200000)
    libgmp.so.10 => /lib/libgmp.so.10 (0x7f8a385f9000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f8a3869f000)

(Tested with versions v6.0.4 and v6.1.0-beta.0)

Azathothas commented 2 months ago

Unfortunately, that Dockerfile you linked doesn't build for me. It errors out, complaining about missing dependencies. Maybe it worked when the PR was originally made, but between then and now, the code has changed significantly that it now doesn't work on the latest version of simplex chat. Perhaps the original author @shumvgolove can help us here.

I have also requested this pkg over at @leleliu008 's https://github.com/leleliu008/ppkg-formula-repository-official-core/issues/6

I am hopeful that there's a way we can compile it statically, however if not, I will probably consider using staticx on it.

Edit: Here's a staticx binary: https://pub.ajam.dev/temp/simplex-chat

🌀 ❯ sha256sum simplex-chat
f66eb32be56a08cb83ddac3c5224110f7a73ee2af4503153816575ef52b30a13  simplex-chat

do let me know if this works as expected

AlbydST commented 2 months ago

This is quite strange, as it appears to build just fine for me. Can you confirm that this fails aswell? If yes, what's the error?

git clone --depth 1 https://github.com/simplex-chat/simplex-chat.git simplex-chat
cd simplex-chat
rm -fv Dockerfile
wget https://raw.githubusercontent.com/simplex-chat/simplex-chat/2abee7edf8cd1c975c9336aa7885f6f818b88d7e/Dockerfile
DOCKER_BUILDKIT=1 docker build .
Azathothas commented 2 months ago

I didn't use the Dockerfile directly, I copied the steps so it would be reproducible for both aarch64 & x86_64 and wouldn't rely on an external source. The script currently (It's broken) is here: https://github.com/Azathothas/Toolpacks/blob/main/.github/scripts/x86_64_Linux/bins/simplex-chat.sh Perhaps I missed some steps or made some silly mistakes. It might have also been that I added some extra ccflags/ldflags. If you can fix that script, It would be much appreciated. Preferably, it would also be much nicer if we don't have to stick to a hardcoded version of ghc/cabal (So I wouldn't have to increase version manually)

Running docker build using the docker file directly:

#/tmp/toolpacks/Uwa4SFh_linux_x86_64 is an empty dir
/tmp/toolpacks/Uwa4SFh_linux_x86_64$ git clone --depth 1 https://github.com/simplex-chat/simplex-chat.git simplex-chat
cd simplex-chat
rm -fv Dockerfile
wget https://raw.githubusercontent.com/simplex-chat/simplex-chat/2abee7edf8cd1c975c9336aa7885f6f818b88d7e/Dockerfile
DOCKER_BUILDKIT=1 docker build .
Cloning into 'simplex-chat'...
remote: Enumerating objects: 2647, done.
remote: Counting objects: 100% (2647/2647), done.
remote: Compressing objects: 100% (2270/2270), done.
remote: Total 2647 (delta 285), reused 1551 (delta 194), pack-reused 0 (from 0)
Receiving objects: 100% (2647/2647), 91.13 MiB | 24.66 MiB/s, done.
Resolving deltas: 100% (285/285), done.
removed 'Dockerfile'
--2024-09-20 04:21:32--  https://raw.githubusercontent.com/simplex-chat/simplex-chat/2abee7edf8cd1c975c9336aa7885f6f818b88d7e/Dockerfile
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1720 (1.7K) [text/plain]
Saving to: 'Dockerfile'

Dockerfile                        100%[===========================================================>]   1.68K  --.-KB/s    in 0s

2024-09-20 04:21:32 (25.9 MB/s) - 'Dockerfile' saved [1720/1720]

[+] Building 521.9s (16/18)                                                                                            docker:default
 => [internal] load build definition from Dockerfile                                                                             0.0s
 => => transferring dockerfile: 1.76kB                                                                                           0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                 0.0s
 => [internal] load .dockerignore                                                                                                0.0s
 => => transferring context: 2B                                                                                                  0.0s
 => [build-stage  1/13] FROM docker.io/library/alpine:latest                                                                     0.0s
 => [internal] load build context                                                                                                1.6s
 => => transferring context: 227.75MB                                                                                            1.5s
 => [build-stage  2/13] RUN apk add --no-cache     curl     git     xz     grep     ghc-dev     gmp-dev     zlib-static     zl  25.8s
 => [build-stage  3/13] RUN curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERAC  122.7s
 => [build-stage  4/13] RUN ghcup set ghc "9.6.3" &&     ghcup set cabal "3.10.1.0"                                              0.5s
 => [build-stage  5/13] RUN cabal install --install-method=copy --installdir=/usr/local/bin hpack-0.36.0                       340.0s
 => [build-stage  6/13] COPY . /project                                                                                          2.2s
 => [build-stage  7/13] WORKDIR /project                                                                                         0.0s
 => [build-stage  8/13] RUN cp ./scripts/cabal.project.local.linux ./cabal.project.local                                         0.2s
 => [build-stage  9/13] RUN sed -i '/- -Wunused-type-patterns/a\  - -O2\n\  - -split-sections\n\  - -with-rtsopts=-N\n\  - -sta  0.3s
 => [build-stage 10/13] RUN hpack                                                                                                0.4s
 => [build-stage 11/13] RUN cabal update                                                                                        25.4s
 => ERROR [build-stage 12/13] RUN cabal build -j exe:simplex-chat                                                                4.4s
------
 > [build-stage 12/13] RUN cabal build -j exe:simplex-chat:
1.258 HEAD is now at aab7b5a support tagging objects with single field
2.256 HEAD is now at 9aa09f1 allow base versions < 4.19
4.033 Cleared directory 'sqlcipher'
4.040 Submodule 'sqlcipher' (https://github.com/sqlcipher/sqlcipher.git) unregistered for path 'sqlcipher'
4.231 HEAD is now at f814ee6 update .gitmodules
4.308 Submodule 'sqlcipher' (https://github.com/sqlcipher/sqlcipher.git) registered for path 'sqlcipher'
4.314 Cloning into '/project/dist-newstyle/src/direct-sq_-1924b92d6e0c4e68/sqlcipher'...
4.314 fatal: could not open '/project/dist-newstyle/src/direct-sq_-1924b92d6e0c4e68/sqlcipher/.git' for writing: File exists
4.315 fatal: clone of 'https://github.com/sqlcipher/sqlcipher.git' into submodule path '/project/dist-newstyle/src/direct-sq_-1924b92d6e0c4e68/sqlcipher' failed
4.315 Failed to clone 'sqlcipher'. Retry scheduled
4.315 BUG: submodule considered for cloning, doesn't need cloning any more?
4.316 fatal: could not get a repository handle for submodule 'sqlcipher'
------
Dockerfile:51
--------------------
  49 |     # Compile simplex-chat
  50 |     RUN cabal update
  51 | >>> RUN cabal build -j exe:simplex-chat
  52 |
  53 |     # Strip the binary from debug symbols to reduce size
--------------------
ERROR: failed to solve: process "/bin/sh -c cabal build -j exe:simplex-chat" did not complete successfully: exit code: 128
leleliu008 commented 2 months ago

I can successfully build it. see this : https://github.com/leleliu008/test/actions/runs/10961122341/job/30437305432

but not a fully statically linked executable.

It should create fully statically linked executables, but cabal has many bugs, I have not found a workaround.

leleliu008 commented 2 months ago

https://github.com/simplex-chat/simplex-chat/pull/3942 this PR actually not produce a fully statically linked execvutable, it's actually dynamically linked.

shumvgolove commented 2 months ago

Hello there!

Thanks for pinging me, I've actually forgot that I've created this PR :^)

Maybe it worked when the PR was originally made, but between then and now, the code has changed significantly that it now doesn't work on the latest version of simplex chat.

To my surprise, it works to this day just fine:

❯ git clone https://github.com/simplex-chat/simplex-chat
Cloning into 'simplex-chat'...
remote: Enumerating objects: 95223, done.
remote: Counting objects: 100% (3075/3075), done.
remote: Compressing objects: 100% (1417/1417), done.
remote: Total 95223 (delta 1779), reused 2454 (delta 1282), pack-reused 92148 (from 1)
Receiving objects: 100% (95223/95223), 238.81 MiB | 26.18 MiB/s, done.
Resolving deltas: 100% (60007/60007), done.
❯ cd simplex-chat
apps  docs      images    scripts  tests    cabal.project  flake.lock  fourmolu.yaml  libsimplex.dll.def  package.yaml  README.md
blog  fastlane  packages  src      website  Dockerfile     flake.nix   install.sh     LICENSE             PRIVACY.md    simplex-chat.cabal
❯ git checkout master
branch 'master' set up to track 'origin/master'.
Switched to a new branch 'master'
❯ curl -LO 'https://raw.githubusercontent.com/simplex-chat/simplex-chat/2abee7edf8cd1c975c9336aa7885f6f818b88d7e/Dockerfile'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1720  100  1720    0     0   4249      0 --:--:-- --:--:-- --:--:--  4246
❯ DOCKER_BUILDKIT=1 docker build --output . .
[+] Building 1212.2s (19/19) FINISHED                                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                                         0.0s
 => => transferring dockerfile: 1.76kB                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                             0.0s
 => [internal] load .dockerignore                                                                                                            0.0s
 => => transferring context: 2B                                                                                                              0.0s
 => CACHED [build-stage  1/13] FROM docker.io/library/alpine:latest                                                                          0.0s
 => [internal] load build context                                                                                                            0.8s
 => => transferring context: 400.25MB                                                                                                        0.8s
 => [build-stage  2/13] RUN apk add --no-cache     curl     git     xz     grep     ghc-dev     gmp-dev     zlib-static     zlib-dev     o  20.1s
 => [build-stage  3/13] RUN curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | BOOTSTRAP_HASKELL_NONINTERACTIVE=1 sh      61.6s 
 => [build-stage  4/13] RUN ghcup set ghc "9.6.3" &&     ghcup set cabal "3.10.1.0"                                                          0.3s 
 => [build-stage  5/13] RUN cabal install --install-method=copy --installdir=/usr/local/bin hpack-0.36.0                                   278.8s 
 => [build-stage  6/13] COPY . /project                                                                                                      0.6s 
 => [build-stage  7/13] WORKDIR /project                                                                                                     0.0s 
 => [build-stage  8/13] RUN cp ./scripts/cabal.project.local.linux ./cabal.project.local                                                     0.1s 
 => [build-stage  9/13] RUN sed -i /- -Wunused-type-patterns/a\  - -O2\n\  - -split-sections\n\  - -with-rtsopts=-N\n\  - -static\n\cc-opt  0.1s 
 => [build-stage 10/13] RUN hpack                                                                                                            0.2s 
 => [build-stage 11/13] RUN cabal update                                                                                                    23.2s 
 => [build-stage 12/13] RUN cabal build -j exe:simplex-chat                                                                                826.3s
 => [build-stage 13/13] RUN bin=$(find /project/dist-newstyle -name "simplex-chat" -type f -executable) &&     mv "$bin" ./ &&     strip ./  0.5s 
 => [export-stage 1/1] COPY --from=build-stage /project/simplex-chat /                                                                       0.1s 
 => exporting to image                                                                                                                       0.1s 
 => => exporting layers                                                                                                                      0.1s 
 => => writing image sha256:72a0aa4d9ab5dd2d330d4fe9a84147af519a11749104dbab8fb0fd802c4b2344                                                 0.0s
 => exporting to client directory                                                                                                            0.1s
 => => copying files 82.47MB                                                                                                                 0.1s

Not sure what's the issue you're getting

https://github.com/simplex-chat/simplex-chat/pull/3942 this PR actually not produce a fully statically linked execvutable, it's actually dynamically linked.

Nope, it does produce the static executable (at least on x86_64).

Running from Alpine Linux:

❯ ldd simplex-chat
/lib/ld-musl-x86_64.so.1: simplex-chat: Not a valid dynamic program
❯ ./simplex-chat -v
SimpleX Chat v6.1.0.3
To update run: curl -o- https://raw.githubusercontent.com/simplex-chat/simplex-chat/master/install.sh | bash

Running from Arch Linux:

[sh ~]$ ldd ./simplex-chat
    not a dynamic executable
[sh ~]$ ./simplex-chat -v
SimpleX Chat v6.1.0.3
To update run: curl -o- https://raw.githubusercontent.com/simplex-chat/simplex-chat/master/install.sh | bash
leleliu008 commented 2 months ago

many options specified via cabal v2-install do not work, but options specified via simplex-chat.cabal works.

This works now: https://github.com/leleliu008/test/actions/runs/10963768364/job/30445930259

Azathothas commented 2 months ago

Thank you, @leleliu008, for figuring it out. Any ETA when the formula/simplex-chat.yml be updated with these changes so it builds? I could repurpose workflow you linked but it again has hard-coded versions, along with patching the package.yaml file to get it to build.

@AlbydST I am against this approach because it's very likely (& as confirmed in my tests), it will eventually stop working, and I don't have the time to come up with these patches.

@shumvgolove I have tried to reproduce your docker file on Debian, Ubuntu and alpine (The three major build environment, this repo uses), all of them fail. Perhaps it's something on my end, as you say it still works and the workflow by @leleliu008 is also almost the same. Along with your PR: https://github.com/simplex-chat/simplex-chat/pull/3942 Mar 22, 2024 There's also already issues asking for the same thing at:

All of these are unresolved, so I had rather wait for the official maintainers themselves to say (preferably do) something about it.

So for now, I think the best compromise is to just add the staticx version, which is pulled straight from official releases simplex-chat-ubuntu-22_04-x86-64 and repacked as a staticx archive. I had already linked and asked if it worked to @AlbydST, but received no answers.

Here's a staticx binary: https://pub.ajam.dev/temp/simplex-chat

🌀 ❯ sha256sum simplex-chat
f66eb32be56a08cb83ddac3c5224110f7a73ee2af4503153816575ef52b30a13  simplex-chat

do let me know if this works as expected

If you would like to reproduce that, here's how:

!#This is copy-pasted from a build-recipe, So please remove sections not relevant to your setup.
#Build (debian-glibc)
pushd "$($TMPDIRS)" >/dev/null 2>&1
 docker stop "debian-builder-unstable" 2>/dev/null ; docker rm "debian-builder-unstable" 2>/dev/null
 docker run --privileged --net="host" --name "debian-builder-unstable" "azathothas/debian-builder-unstable:latest" \
  bash -l -c '
  #Setup ENV
   mkdir -p "/build-bins" && pushd "$(mktemp -d)" >/dev/null 2>&1
   sudo apt-get update -y -qq
   sudo apt-get install -y -qq libpcre3 libgmp3-dev xdg-utils zlib1g-dev
   sudo apt-get update -y -qq
  #Fetch
   eget "https://github.com/simplex-chat/simplex-chat" --asset "chat" --asset "ubuntu-22_04" --asset "x86_64" --asset "^desktop" --asset "^deb" --to "./simplex-chat"
   staticx --loglevel DEBUG "./simplex-chat" --strip "/build-bins/simplex-chat"
  #strip & info 
   objcopy --remove-section=".comment" --remove-section=".note.*" "/build-bins/simplex-chat"
   strip --strip-debug --strip-dwo --strip-unneeded -R ".comment" -R ".gnu.version" "/build-bins/simplex-chat"
   file "/build-bins/simplex-chat" && du -sh "/build-bins/simplex-chat"
   popd >/dev/null 2>&1
  '
#Copy & Meta
 docker cp "debian-builder-unstable:/build-bins/." "$(pwd)/"
 find "." -maxdepth 1 -type f -exec file -i "{}" \; | grep "application/.*executable" | cut -d":" -f1 | xargs realpath
 #Meta
 find "." -maxdepth 1 -type f -exec sh -c 'file "{}"; du -sh "{}"' \;
 sudo rsync -av --copy-links --exclude="*/" "./." "$BINDIR"
#Delete Containers
 docker stop "debian-builder-unstable" 2>/dev/null ; docker rm "debian-builder-unstable"
 popd >/dev/null 2>&1

The tldr is, I would need a script that can build simplex-chat cli reliably, without too many patches and that's guaranteed not to break with updates. If that's not possible, then staticx or even dockerc variant is what I think should be added.

leleliu008 commented 2 months ago

sed -i '/- -Wunused-type-patterns/a\ - -O2\n\ - -split-sections\n\ - -with-rtsopts=-N\n\ - -static\n\cc-options: -static\n\ld-options: -static -pthread' package.yaml

this line is the most important patch, it modify the package.yaml, then use hpack command regenerate simplex-chat.cabal, cabal reads simplex-chat.cabal then pass options to ghc. Theoretically, these options should be supplied via cabal v2-install command , Actually, it does not work. I did not try other command like cabal build cabal install

leleliu008 commented 2 months ago

https://github.com/leleliu008/ppkg-package-manually-build/actions/runs/10973554214

xplshn commented 2 weeks ago

Wow @leleliu008, you always deliver :) Keep up the good work <3