search

tamatebako / tebako

Tebako: an executable packager (for Ruby programs)
https://www.tebako.org
56 stars 8 forks source link

How to package an application for which a gem already exists ? #207

Closed laurent-martin closed 1 month ago

laurent-martin commented 1 month ago

Hi,

I am trying to press the ruby script ascli included in gem https://rubygems.org/gems/aspera-cli on macos (sequoia).

I have read tebako documentation several times, I have tried many times to build , using various techniques, but that was unfruitful.

The table "Packaging scenarios" is useful, but somehow it lacks command line examples.

So the scenario: a tool is already packaged in a ruby gem, how to press it , on host (macos, windows, Linux) ?

I want to avoid cloning a repo, I wan only to get the released gem with all its dependencies in one executable.

For example, I tried this:

gem fetch aspera-cli:4.19.0
gem unpack aspera-cli-4.19.0.gem
tebako press --Ruby=3.3.4 --root=$PWD/aspera-cli-4.19.0 --entry-point=bin/ascli --output=toto

But dependencies are missing.

maxirmx commented 1 month ago

Hi, @laurent-martin

Thank you for your question. Sample output follows, remarks will be in the next comment

maxirmx@MSS-WS-N:~/Projects/tebako$ mkdir test
maxirmx@MSS-WS-N:~/Projects/tebako$ cd test
maxirmx@MSS-WS-N:~/Projects/tebako/test$ gem fetch aspera-cli:4.19.0
Fetching aspera-cli-4.19.0.gem
Downloaded aspera-cli-4.19.0
maxirmx@MSS-WS-N:~/Projects/tebako/test$ cd ..
maxirmx@MSS-WS-N:~/Projects/tebako$ tebako press -r test -e ascli -o ascli.tebako
Tebako executable packager version 0.9.1
Running tebako press at /home/maxirmx/Projects/tebako
   Ruby version:              '3.2.5'
   Project root:              '/home/maxirmx/Projects/tebako/test'
   Application entry point:   'ascli'
   Package file name:         '/home/maxirmx/Projects/tebako/ascli.tebako'
   Loging level:              'error'
   Package working directory: '<Host current directory>'
Running tebako press script
Packaging /home/maxirmx/Projects/tebako/test folder with ascli entry point to /home/maxirmx/Projects/tebako/ascli.tebako
-- OSTYPE: 'linux-gnu'
-- NCORES: 16
Configuration summary:
-- ruby: v3.2.5 at /home/maxirmx/Projects/tebako/deps/src/_ruby_3.2.5
-- dwarfs with tebako wrapper: @v0.7.1 at /home/maxirmx/Projects/tebako/deps/src/_dwarfs_wr
-- DATA_SRC_DIR: /home/maxirmx/Projects/tebako/o/s
-- DATA_PRE_DIR: /home/maxirmx/Projects/tebako/o/r
-- DATA_BIN_DIR: /home/maxirmx/Projects/tebako/o/p
-- DATA_BIN_FILE: /home/maxirmx/Projects/tebako/o/p/fs.bin
-- Target binary directory: /home/maxirmx/Projects/tebako/o/s/bin
-- Target library directory: /home/maxirmx/Projects/tebako/o/s/lib
-- Target local directory: /home/maxirmx/Projects/tebako/o/s/local
-- Target Gem directory:: /home/maxirmx/Projects/tebako/o/s/lib/ruby/gems/3.2.0
-- FS_MOUNT_POINT: /__tebako_memfs__
-- Building for Win32 Ruby (RB_W32): OFF
-- Removing GLIBC_PRIVATE reference: ON
-- Not building Ruby extensions: dbm,win32,win32ole,-test-/*
-- Ruby build cflags='-fPIC -I/home/maxirmx/Projects/tebako/deps/include -I/home/maxirmx/.rbenv/versions/3.2.4/lib/ruby/gems/3.2.0/gems/tebako-0.9.1/include'
-- Ruby build LDFLAGS='-L/home/maxirmx/Projects/tebako/deps/lib -L/home/maxirmx/Projects/tebako/o'
-- openssl Ruby option=''
-- libyaml Ruby option=''
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/maxirmx/Projects/tebako/o
[ 53%] Built target _patchelf
[ 60%] Built target _dwarfs_wr
[ 89%] Built target _ruby_3.2.5
Tebako setup has been verified
[ 89%] Built target setup
-- Running init script
   ... creating packaging environment at /home/maxirmx/Projects/tebako/o/s
-- Running deploy script
'tebako-runtime' was not loaded.
RubyGems Environment:
  - RUBYGEMS VERSION: 3.4.19
  - RUBY VERSION: 3.2.5 (2024-07-26 patchlevel 208) [x86_64-linux]
  - INSTALLATION DIRECTORY: /home/maxirmx/Projects/tebako/o/s/lib/ruby/gems/3.2.0
  - USER INSTALLATION DIRECTORY: /home/maxirmx/.gem/ruby/3.2.0
  - RUBY EXECUTABLE: /home/maxirmx/Projects/tebako/o/s/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /home/maxirmx/Projects/tebako/o/s/bin
  - SPEC CACHE DIRECTORY: /home/maxirmx/Projects/tebako/o/s/spec_cache
  - SYSTEM CONFIGURATION DIRECTORY: /home/maxirmx/Projects/tebako/o/s/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /home/maxirmx/Projects/tebako/o/s/lib/ruby/gems/3.2.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /home/maxirmx/.rbenv/versions/3.2.4/bin
     - /home/maxirmx/.rbenv/libexec
     - /home/maxirmx/.rbenv/plugins/ruby-build/bin
     - /home/maxirmx/.rbenv/shims
     - /home/maxirmx/.rbenv/bin
     - /home/maxirmx/.vscode-server/bin/384ff7382de624fb94dbaf6da11977bba1ecd427/bin/remote-cli
     - /home/maxirmx/.nix-profile/bin
     - /home/maxirmx/.cargo/bin
     - /home/maxirmx/.local/bin
     - /home/maxirmx/.nvm/versions/node/v18.18.0/bin
     - /home/maxirmx/.rbenv/shims
     - /home/maxirmx/.rbenv/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
     - /usr/games
     - /usr/local/games
     - /usr/lib/wsl/lib
     - /mnt/c/Program Files/Common Files/Oracle/Java/javapath
     - /mnt/c/Windows/system32
     - /mnt/c/Windows
     - /mnt/c/Windows/System32/Wbem
     - /mnt/c/Windows/System32/WindowsPowerShell/v1.0/
     - /mnt/c/Windows/System32/OpenSSH/
     - /mnt/c/Program Files/7-Zip
     - /mnt/c/ProgramData/chocolatey/bin
     - /mnt/c/Program Files/dotnet/
     - /mnt/c/Program Files/Microsoft SQL Server/130/Tools/Binn/
     - /mnt/c/Program Files/CMake/bin
     - /mnt/c/Program Files (x86)/GnuPG/bin
     - /mnt/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/
     - /mnt/c/Program Files/Microsoft SQL Server/150/Tools/Binn/
     - /mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn/
     - /mnt/c/Program Files/PowerShell/7/
     - /mnt/c/Program Files/Git/cmd
     - /mnt/c/Program Files/Docker/Docker/resources/bin
     - /mnt/c/Ruby31-x64/bin
     - /mnt/c/Users/Maxim/AppData/Local/Programs/Python/Python311/Scripts/
     - /mnt/c/Users/Maxim/AppData/Local/Programs/Python/Python311/
     - /mnt/c/vcpkg/installed/x64-windows/bin
     - /mnt/c/Users/Maxim/AppData/Local/Microsoft/WindowsApps
     - /mnt/c/Users/Maxim/AppData/Local/GitHubDesktop/bin
     - /mnt/c/Users/Maxim/AppData/Local/Programs/Microsoft VS Code/bin
     - /mnt/c/Users/Maxim/.dotnet/tools
     - /snap/bin
   ... installing tebako-runtime gem
   ... @ /home/maxirmx/Projects/tebako/o/s/bin/gem install tebako-runtime --no-document --install-dir /home/maxirmx/Projects/tebako/o/s/lib/ruby/gems/3.2.0
   ... installing Ruby gem from /home/maxirmx/Projects/tebako/test/aspera-cli-4.19.0.gem
   ... installing /home/maxirmx/Projects/tebako/test/aspera-cli-4.19.0.gem gem
   ... @ /home/maxirmx/Projects/tebako/o/s/bin/gem install /home/maxirmx/Projects/tebako/test/aspera-cli-4.19.0.gem --no-document --install-dir /home/maxirmx/Projects/tebako/o/s/lib/ruby/gems/3.2.0
   ... target entry point will be at /__tebako_memfs__/bin/ascli
   ... stripping the output
[ 89%] Built target source_filesystem
I 17:16:05.920845 scanning "/home/maxirmx/Projects/tebako/o/s"
I 17:16:06.015153 assigning directory and link inodes...
I 17:16:06.015431 waiting for background scanners...
I 17:16:06.015474 scanning CPU time: 293.5ms
I 17:16:06.015487 finalizing file inodes...
I 17:16:06.016006 saved 350.5 KiB / 23.66 MiB in 82/3116 duplicate files
I 17:16:06.016042 assigning device inodes...
I 17:16:06.016099 assigning pipe/socket inodes...
I 17:16:06.016151 building metadata...
I 17:16:06.016191 building blocks...
I 17:16:06.016200 saving names and symlinks...
I 17:16:06.016232 using a 4 KiB window at 512 B steps for segment analysis
I 17:16:06.016320 bloom filter size: 64 KiB
I 17:16:06.016517 ordering 3034 inodes using nilsimsa similarity...
I 17:16:06.016637 nilsimsa: depth=20000 (1000), limit=255
I 17:16:06.017195 updating name and link indices...
I 17:16:06.017681 pre-sorted index (967 name, 26 path lookups) [1.013ms]
I 17:16:06.080525 3034 inodes ordered [63.95ms, 63.96ms CPU]
I 17:16:06.080697 waiting for segmenting/blockifying to finish...
I 17:16:06.211445 segmenting/blockifying CPU time: 193.6ms
I 17:16:06.211690 bloom filter reject rate: 97.178% (TPR=0.020%, lookups=17345766)
I 17:16:06.211711 segmentation matches: good=45, bad=52, total=97
I 17:16:06.211738 segmentation collisions: L1=0.004%, L2=0.002% [46659 hashes]
I 17:16:06.211765 saving chunks...
I 17:16:06.212273 saving directories...
I 17:16:06.213213 saving shared files table...
I 17:16:06.215987 saving names table... [2.618ms]
I 17:16:06.216010 saving symlinks table... [1.01us]
I 17:16:06.217618 waiting for compression to finish...
I 17:16:11.347633 compressed 23.66 MiB to 6.604 MiB (ratio=0.279162)
I 17:16:11.348917 compression CPU time: 7.3s
I 17:16:11.348963 filesystem created without errors [5.428s]
737 dirs, 0/0 soft/hard links, 3116/3116 files, 0 other
original size: 23.66 MiB, scanned: 23.31 MiB, hashed: 2.188 MiB (1248 files)
saved by deduplication: 350.5 KiB (82 files), saved by segmenting: 536.1 KiB
filesystem: 22.79 MiB in 2 blocks (3108 chunks, 3034/3034 inodes)
compressed filesystem: 2 blocks/6.604 MiB written [depth: 20000]
[ 89%] Built target packaged_filesystem
[ 92%] Building CXX object CMakeFiles/tebako-fs.dir/home/maxirmx/Projects/tebako/deps/src/tebako/tebako-fs.cpp.o
[ 96%] Linking CXX static library libtebako-fs.a
[100%] Built target tebako-fs
   ... building tebako package
   ... @ make ruby -j16
   ... @ make -j16
   ... @ /home/maxirmx/Projects/tebako/deps/bin/patchelf --remove-needed-version libpthread.so.0 GLIBC_PRIVATE /home/maxirmx/Projects/tebako/deps/src/_ruby_3.2.5/ruby
Created tebako package at "/home/maxirmx/Projects/tebako/ascli.tebako"
[100%] Built target tebako
maxirmx@MSS-WS-N:~/Projects/tebako$ ./ascli.tebako
W, [2024-10-24T20:16:53.569227 #30878]  WARN -- : No config file found. New configuration file: /home/maxirmx/.aspera/ascli/config.yaml
ERROR: Argument: missing argument: command (Symbol)
Use:
- alee
- aoc
- ats
- config
- console
- cos
- faspex
- faspex5
- faspio
- help
- httpgw
- node
- orchestrator
- preview
- server
- shares
Use option -h to get help.
maxirmx commented 1 month ago

Remarks:

  1. Do not unpack the gem, please
  2. Do not provide the path to the executable, i.e.: --entry-point=ascli and not --entry-point=bin/ascli (I know, it is weird, but tebako command line was created compatible with ruby-packer and it is the rudiment)
maxirmx commented 1 month ago

@laurent-martin I will be happy to help if there are other questions or issues. You are right about scenatious / samples, thank you for the suggestion.

laurent-martin commented 1 month ago

Hi,

Thanks, finally, I have it compile on macos Sequoia.

I have to tweak a little the PATH so that tools are found, and the right ar is used for clang ...

Also, I find that it might be better if some of the libs were installed otherwise than in the OS (brew). I compile other projects, where I use a more recent version of boost, which I get with conan, but if boost is installed with brew, then I have a clash, so I have to install/uninstall ... (or I should look how to avoid that more easily maybe).

Also, I have users on RHEL 7 and 8, and the patchelf still does not fix the problem of glibc for older OSes, that's why also I look at building myself.

https://github.com/IBM/aspera-cli/blob/develop/examples/build_exec

Thanks again.

ronaldtse commented 1 month ago

Thank you @laurent-martin for the update!

We are always seeking ways to simplify usage for Tebako users, and clearly we have lots to learn in your use case.

The current setup process is a bit cumbersome:

Furthermore, RHEL 7/8 is an interesting use case, and this is definitely not what we want users to go through:

@maxirmx thoughts?

maxirmx commented 1 month ago

RHEL 7/8

In short, the minimal GLIBC version supported by the Tebako package is the GLIBC version on the system where the package was created. While patchelf enables forward compatibility, it does not ensure backward compatibility.

Currently, we provide an Ubuntu 20 container, which requires a minimum GLIBC version of 2.31. By default RHEL7 has GLIBC 2.17, RHEL8 has GLIBC 2.28.

If the question is, "Is it possible to create a Tebako container that produces GLIBC 2.17-compatible binaries?" the answer is "probably yes". To achieve this, we would need to base the build environment on RHEL7 and address any compatibility issues with folly and dwarfs, which may take a few months.

maxirmx commented 1 month ago

MacOS/brew

It is possible to have two brew instances. Discussion is here https://github.com/orgs/Homebrew/discussions/2843 / https://stackoverflow.com/questions/70821136/can-i-install-arm64-libraries-on-x86-64-with-homebrew/70822921#70822921 although it covers a different case

The other option is to create packaging container that will generate MacOS binaries like rake-compiler-dock does but it is also a big effort