= Tebako: An advanced image packager for interpretive languages
Platform tests on GitHub: image:https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml/badge.svg["Ubuntu amd64", link="https://github.com/tamatebako/tebako/actions/workflows/ubuntu.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/alpine.yml/badge.svg["Alpine", link="https://github.com/tamatebako/tebako/actions/workflows/alpine.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/macos.yml/badge.svg["macOS amd64", link="https://github.com/tamatebako/tebako/actions/workflows/macos.yml"] image:https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml/badge.svg["Windows msys", link="https://github.com/tamatebako/tebako/actions/workflows/windows-msys.yml"]
Platform tests on Cirrus: image:https://api.cirrus-ci.com/github/tamatebako/tebako.svg?branch=main&task=ubuntu-aarch64["Ubuntu aarch64", link="https://cirrus-ci.com/github/tamatebako/tebako"]
Tools tests on GitHub: image:https://github.com/tamatebako/tebako-ci-containers/actions/workflows/build-containers.yml/badge.svg["Tebako cobtainers", link="https://github.com/tamatebako/tebako-ci-containers/actions/workflows/build-containers.yml"]
Quality: image:https://github.com/tamatebako/tebako/actions/workflows/lint-and-rspec.yml/badge.svg["lint and rspec", link="https://github.com/tamatebako/tebako/actions/workflows/lint-and-rspec.yml"] image:https://codecov.io/gh/tamatebako/tebako/graph/badge.svg?token=XD3emQ5qsY["Tebako cli rspec coverage", link="https://codecov.io/gh/tamatebako/tebako"]
== Purpose
Tebako is an advanced executable packager designed for applications written in interpretive languages.
It simplifies distribution and deployment by packaging your entire project with a bundled runtime into a single, performant, executable binary.
== Architecture
A Tebako-packaged binary is effectively a self-executing container-in-a-file.
The packaged binary contains the following components:
An on-file filesystem (OFFS) containing all the project files and dependencies in DwarFS format.
A runtime environment that includes the necessary libraries and interpreters, with patched filesystem calls that redirect access of project files to the on-file filesystem.
An executable loader that loads the on-file filesystem in memory and executes the project.
== Supported runtimes, platforms and architectures
Tebako artifacts can be built and executed on the following platforms and architectures.
.Supported platforms and architectures [cols="3", options="header"] |=== | Platform and version | Architectures | Build system
3+| Linux | Ubuntu 20.04 | amd64, aarch64 | gcc/g++: 10; clang/clang++: 12 | Alpine 3.17 | amd64 | gcc/g++: default; clang/clang++: default
3+| macOS | macOS 12 (Monterey) | amd64 | xcode: [13.1, 14.3.1] | macOS 13 (Ventura) | amd64, aarch64 | xcode: [13.1, 14.3.1] | macOS 14 (Sonoma) | amd64, aarch64 | xcode: [13.1, 14.3.1, 15.4]
3+| Windows | Windows 10 | amd64 | MinGW ucrt64 | Windows 11 | amd64 | MinGW ucrt64 | Windows Server 2019 | amd64 | MinGW ucrt64 | Windows Server 2022 | amd64 | MinGW ucrt64
|===
Windows build caveats:
Tebako may face errors related to CMake path length limitations (https://gitlab.kitware.com/cmake/cmake/-/issues/25936). This error may affect not tebako itself but the gems that need to be package and use CMake to build native extensions. There is no workaround for this issue as it looks like is a limitation of the manifest used to build CMake executable.
.Supported Ruby versions [cols="2", options="header"] |=== | Ruby version | Supported platforms
| 2.7.8 | Linux, macOS | 3.0.7 | Linux, macOS | 3.1.6 | Linux, macOS, Windows | 3.2.{4,5} | Linux, macOS, Windows | 3.3.{3,4,5} | Linux, macOS, Windows
|===
NOTE: Our goal is to support all maintained Ruby releases, including minor versions.
== Package portability
=== General
Tebako packages are designed to be "forward portable" across different operating systems and architectures to allow for easy distribution and deployment.
Forward portability means that a package created on a specific platform can be executed on a newer version of the same platform.
=== macOS
macOS packages are forward portable across different macOS versions.
[example] A Tebako executable package built on macOS 12 (Monterey) can be executed on macOS 14 (Sonoma), but not vice versa.
x86_64 macOS packages can be run on Apple M (ARM) systems.
=== Linux distributions using musl
Packages built for the https://musl.libc.org[`musl` implementation of the C standard library] (such as https://alpinelinux.org[Alpine Linux]) are forward portable.
[example] A Tebako executable package built on Alpine 3.17 can be executed on Alpine 3.19.
Usage of the https://github.com/tamatebako/tebako-ci-containers[Tebako Docker containers] for packaging is encouraged since it eliminates the effort needed for toolchain setup and configuration.
=== Linux distributions using glibc
Packages built for the
https://sourceware.org/glibc[`glibcimplementation of the C standard library] are forward portable if the--patchelf` experimental option is enabled.
The --patchelf option allows these packages to be portable to Linux GNU
distributions with GLIBC version 2.31 and above.
[example]
A Tebako executable package built on Ubuntu 20.04 with --patchelf option can
be executed on Rocky Linux 9.
Usage of the https://github.com/tamatebako/tebako-ci-containers[Tebako Docker containers] for packaging is encouraged since it eliminates the effort needed for toolchain setup and configuration.
.Minimum versions of GLIBC Linux distributions that support Tebako packages with forward portability [cols="3", options="header"] |=== | Distribution | Minimal supported version | GLIBC version
| Ubuntu | 20.04 (Focal Fossa) | GLIBC 2.31 | Debian | 11 (Bullseye) | GLIBC 2.31 | Rocky Linux | 9 | GLIBC 2.34 | Fedora | 33 | GLIBC 2.32 | CentOS | 9 | GLIBC 2.34 | Red Hat Enterprise Linux (RHEL) | 9 | GLIBC 2.34 | Oracle Linux | 9 | GLIBC 2.34
|===
== Future plans
== FAQ
=== Why use Tebako?
Tebako is particularly useful for developers who need to:
=== How do I know I need Tebako?
You might need Tebako if you:
=== What is DwarFS?
https://github.com/mhx/dwarfs[DwarFS] is a fast, high compression read-only user-land file system designed for efficient storage and access of large collections of files.
It is used by Tebako to package applications into a compact and efficient format.
Tebako uses https://github.com/tamatebako/libdwarfs[libdwarfs], the library form of https://github.com/mhx/dwarfs[DwarFS], developed for the Tebako project.
=== When is Tebako better than comparable solutions?
Tebako offers several advantages over comparable solutions for supported interpretive languages.
They are listed in order of the degree of virtualization below.
Tebako stands out by providing a lightweight runtime bundling approach that simplifies distribution and deployment while offering flexibility and efficiency.
It eliminates the need for users to have specific runtimes installed and ensures consistency across different environments.
With Tebako, you can package your entire project with a bundled runtime into a single, performant, executable binary.
[cols="a,3a,3a"] |=== | Solution | Pros | Cons
| Virtual machines (VMs) |
|---|
| Docker |
|---|
| Tebako |
|---|
| Ruby Gems |
|---|
|===
== Usage
=== Command-line interface
Tebako works by packaging your project into a single executable binary that includes all the necessary dependencies.
The way to work with Tebako is through its command-line interface (CLI). It provides the following commands:
setup::
Prepares the Tebako packaging environment.
press::
Packages a project into a single executable binary.
clean::
Removes Tebako artifacts.
clean_ruby::
Removes Tebako Ruby artifacts.
hash::
Calculates the Tebako script hash for use as a cache key in CI/CD environments.
extract::
Extracts the filesystem from a Tebako package.
version::
Displays the Tebako version.
help::
Displays the help message.
== Usage
=== General
Tebako can be used in two ways:
Please refer to the <
[[installation]] == Installation
=== General
Installation of Tebako is only needed in order to package an application.
There is no need to install anything for users who run the packaged application.
=== Using Docker
==== General
If you have Docker installed and available, the easiest way to run Tebako is through the official Docker containers.
Docker containers with preinstalled Tebako packaging environments for Ubuntu and Alpine Linux are available at https://github.com/tamatebako/tebako-ci-containers[tebako-ci-containers].
==== Pull the container
Pull the Tebako container image.
<container_tag>:: is the desired image tag (e.g., ubuntu-20.04 or alpine-3.17).
==== Running Tebako commands in the container
Simply prefix the Tebako command with docker run and the container image.
==== Packaging from outside the container
To package your application from outside the container, just run a single Docker command.
This command mounts the application folder into the container and runs the
tebako press command, specifying the application root, entry point, output
location, and Ruby version.
<application_folder>:: is the path to your application folder.
<container_tag>:: is the desired image tag (e.g., ubuntu-20.04 or alpine-3.17).
Assume that you have a Ruby application in the fontist folder of the current
directory.
You can package it to ./fontist-package using the following command:
====
==== Packaging from inside the container
It is also possible to package an application from inside the Tebako container.
Start and enter the container interactively.
<application_folder>:: is the path to your application folder.
<container_tag>:: is the desired image tag (e.g., ubuntu-20.04 or alpine-3.17).
Once inside, run the tebako press command:
Assume that you have a Ruby application in the fontist folder of the current
directory.
You can package it to ./fontist-package using the following command:
$ docker run -it --rm -v $PWD:/mnt/w ghcr.io/tamatebako/tebako-
====
=== Local installation
==== General
There are cases where Docker may not be suitable for your needs, such as:
. Admin privileges: Running Docker requires administrative privileges, which means Docker may not be available to users on their machines.
. Performance penalty: Docker introduces a performance penalty due to the overhead of running containers. This can be a concern when packaging complex applications that require heavy memory usage.
In such cases, you can choose to install Tebako locally.
Tebako is distributed as a Ruby gem. A Ruby environment is necessary.
==== Prerequisites
These prerequisites are needed only for users who want to install Tebako on their machine and build all Tebako components locally.
If you use Docker, there is no need to set up these prerequisites.
===== Ubuntu 20.04
====== General
There are several prerequisites that need to be installed on Ubuntu 20.04 for Tebako to work correctly.
====== GNU C/C++ 10+ or Clang C/C++ 12+
or
====== CMake version 3.20+
Tebako requires CMake at a version of at least 3.20+.
If such CMake version is not available as a default package, set it up as follows.
====== Other development tools and libraries
===== Alpine 3.17
====== General
There are several prerequisites that need to be installed on Alpine 3.17 for Tebako to work correctly.
====== APK packages
Run the following command to install all prerequisites.
===== macOS
====== General
There are several prerequisites that need to be installed on macOS for Tebako to work correctly.
The following instructions work for:
====== Homebrew packages
We use Homebrew to install the necessary packages on macOS.
====== Bison 3+
Tebako requires Bison 3+.
On macOS 14, the default Bison version is 2.3, and the Homebrew formula is keg-only, which means that the full path to the Bison binary must be used to utilize the correct version.
Run the following command prior to using Tebako, or add it into your shell profile.
====== jemalloc Library Build
The libdwarfs build script creates an additional jemalloc installation on macOS. This is done to satisfy the magic applied by folly during linking but uses a static library.
If the library is created in an emulated environment (QEMU, Rosetta, etc.), there are known issues (link:https://github.com/jemalloc/jemalloc/issues/1997[jemalloc issue #1997]) where jemalloc incorrectly defines the number of significant virtual address bits (lg-vaddr parameter).
These issues can be fixed by explicitly setting the --with-lg-vaddr parameter for the jemalloc build. We decided not to automate this since we do not feel that we can provide reasonable test coverage. Instead, our build script accepts the LG_VADDR environment variable and passes it to the jemalloc build as --with-lg-vaddr=${LG_VADDR}.
The LG_VADDR parameter specifies the number of significant virtual address bits, which can vary based on the CPU architecture and emulation status.
Simple script to set LG_VADDR. Please note that it is provided for illustration only.
ARCH=$(uname -m)
if [[ "$ARCH" == "x86_64" && $(sysctl -n sysctl.proc_translated) == "1" ]]; then echo "Running on Apple Silicon under Rosetta 2 emulation" export LG_VADDR=39 elif [[ "$ARCH" == "arm64" ]]; then echo "Running on Apple Silicon" export LG_VADDR=39 else echo "Running on Intel Silicon" export LG_VADDR=48 fi
===== Windows
====== General
There are several prerequisites that need to be installed on macOS for Tebako to work correctly.
The following instructions work for:
====== Ruby
To run Tebako you need to have Ruby installed. It is simplest to use the Ruby development environment provided by https://rubyinstaller.org[RubyInstaller].
For example, Ruby+Devkit 3.1.4-1.
====== MinGW ucrt64
Enable MinGW ucrt64 and install the necessary packages.
The ridk command originates from the RubyInstaller installation.
== Packaging
=== Tebako root folder (aka prefix) selection
The Tebako prefix determines the base directory for the Tebako setup.
It is an essential part of configuring how Tebako operates within your system.
The selection of the Tebako prefix follows a specific order of precedence to ensure flexibility and ease of use:
. User-specified prefix: The most direct way to set the root folder is by specifying it through a command-line argument.
. Current Working Directory (PWD):
If the prefix option is explicitly set to PWD, Tebako uses the current working
directory as Tebako root folder.
. Environment variable (TEBAKO_PREFIX):
In the absence of a user-specified option, Tebako looks for an environment
variable named TEBAKO_PREFIX. If found, its value is used as the root folder.
. Default value:
If no prefix is specified and the TEBAKO_DIR environment variable is not set,
Tebako defaults to using a directory named .tebako in the user's home
directory.
Path Expansion: Regardless of the method used to set the Tebako prefix, Tebako
expands the provided path to an absolute path. This expansion includes resolving
relative paths based on the current working directory and expanding user
directory shortcuts like ~.
=== Commands
Tebako provides several commands to manage the packaging and deployment process.
==== Press
This command "presses" a Ruby project using the Tebako components built in the Tebako
root folder (<tebako-root-folder>).
press command can take up to an hour as it sets up
the packaging environment and collects the required dependencies. Subsequent
invocations are much faster.Upon the next invocation tebako will use previously created packaging environment. The press process itself takes minutes.
You can manage setup of packaging environment manually; please refer to description of setup and clean commands below.
Where:
<tebako-root-folder>::
the Tebako root folder (see details in the Tebako Root Folder Selection section)
Ruby::
this parameter defines Ruby version that will be packaged (optional, defaults to
3.1.6)
project-root::
a folder at the host source file system where project files are located
entry-point::
an executable file (binary executable or script) that shall be started when
packaged file is called
output::
the output file name (optional, defaults to <current folder>/<entry point base name>)
log-level::
logging level for the Tebako built-in memory filesystem driver
(optional, defaults to error)
cwd::
a folder within Tebako memfs where the packaged application will start. This folder should be specified relative to the memfs root.
If not provided, the application will start within the current folder of the host (i.e., at $PWD).
This option is required because it is not possible to change the directory to a memfs folder until the package is started, as opposed to any host folder
that can be set as the current directory before Tebako package invocation. Tebako saves original working directory in a global Ruby variable $tebako_original_pwd.
devmode:: flag that activates development mode, in which Tebako's cache and
packaging consistency checks are relaxed.
patchelf::
flag that removal a reference to GLIBC_PRIVATE version of libpthread from tebako package. This allows Linux Gnu packages to run against versions of
libpthread that differ from the version used for packaging. For example, package created at Ubuntu 20 system can be used on Ubuntu 22. This option makes
sense and works on Gnu Linux only. The feature is exeprimental, we may consider other approach in the future.
tebafile::
the tebako configuration file (optional, defaults to $PWD/.tebako.yml).
Please refer to the separate section below for tebafile description.
+
NOTES:
entry-point and project-root-folder are required parameters and may be provided either via command-line or in tebafile.====
==== Setup
This command sets up the Tebako packaging environment.
Collects required packages, builds the and creates packaging environment. This is a lengthy task that can take significant time, up to 1 hour.
Tebako supports several configurations at a single system given that their root directories differ and multiple Ruby versions within single configuration
This command is optional, tebako creates packaging environment automatically upon the first invocation of press command.
However, if you plan to use tebako in CI/CD environment with caching it is
highly recommended to build cache based on tebako setup output. Building cache
based on tebako press may create inconsistent environment upon restore.
Where:
<tebako-root-folder>:: the Tebako root folder (see details in the Tebako Root Folder Selection section)
Ruby:: parameter defines Ruby version that will be packaged (optional, defaults to 3.1.6)
tebafile::
the tebako configuration file (optional, defaults to $PWD/.tebako.yml).
Please refer to the separate section below for tebafile description.
devmode:: flag activates development mode, in which Tebako's cache and packaging consistency checks are relaxed.
Please note that this mode is not intended for production use and should only be used during development.
==== Clean
This command cleans up all Tebako artifacts in the specified prefix directory.
NOTE: These artifacts are created by the setup and press commands.
Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.
Where:
<tebako-root-folder>:: the Tebako root folder (see details in the Tebako Root Folder Selection section)
tebafile::
the tebako configuration file (optional, defaults to $PWD/.tebako.yml).
Please refer to the separate section below for tebafile description.
====
==== Clean Ruby
This command cleans up only the Ruby artifacts from the specified prefix directory.
NOTE: These artifacts are created by the setup and press commands.
Normally you do not need to do it, since Tebako packager optimizes artifacts
lifecycle on its own.
NOTE: Compiled DwarFS libraries are not cleaned.
$ tebako clean_ruby
[-p|--prefix=
Where:
<tebako-root-folder>::
the Tebako setup folder (optional, defaults to current folder)
Ruby::
defines Ruby version that will cleaned (optional, cleans all versions by default)
tebafile::
the tebako configuration file (optional, defaults to $PWD/.tebako.yml).
Please refer to the separate section below for tebafile description.
====
==== Build script hash
This command outputs a hash value for the Tebako build script, which can be used as a cache key in CI/CD pipelines.
=== Tebako configuration file
It is possible to provide all or some options for the tebako setup/press/clean/clean_ruby commands via Tebako configuration file ('tebafile').
Tebafile is a YAML file with a single section 'options'. The options are the same as long names for the command line. For, example for the prefix option
the key in the YAML file would be 'prefix'.
Please note that the options provided on the command line have preference over tebafile settings.
=== Exit codes
The Tebako CLI exits with different exit codes to indicate the status of the operation. The following table lists the possible exit codes and their meanings.
.Tebako CLI exit codes [cols="a,a"] |=== | Code | Condition
| 0 | No error
| 1 | Invalid command line
| 101 | tebako setup failed at configuration step
| 102 | tebako setup failed at build step
| 103 | tebako press failed at configuration step
| 104 | tebako press failed at build step
| 253 | Unsupported Ruby version
| 254 | Unsupported operating systems
| 255 | Internal error
|===
== Packaging scenarios with Ruby
Tebako for Ruby supports the following packaging scenarios.
This is high-level description of the Tebako Ruby packaging mechanism.
NOTE: These scenarios were inspired by the ruby-packer approach.
NOTE: Tebako Ruby is created independently from ruby-packer, no line of code
was copied from ruby-packer.
Depending on the configuration files that are present in the root project folder, the Tebako Ruby packager supports different packaging scenarios.
These scenarios differ in what files are packaged and where the entry point is located.
Here is a summary of the scenarios:
[cols="a,2a,4a,3a,a,a,a"]
|===
| Scenario | Description | Packaging | Entry point | *.gemspec | Gemfile | *.gem
| 1
| Simple ruby script
| Copy <project-root> with all sub-folders to packaged filesystem
| <mount_point>/local/<entry_point base name>
| No
| No
| No
| 2
| Packaged gem
| Install the gem with gem install to packaged filesystem
| <mount_point>/bin/<entry_point base name> (i.e., binstub is expected)
| No
| No
| One
| 3 | Gem source, no bundler |
|---|
. Build the gem using gem build command at the host
. Install it with gem install to packaged filesystem
| <mount_point>/bin/<entry_point base name> (i.e., binstub is expected)
| One
| No
| Any
| 4 | Gem source, bundler |
|---|
. Collect dependencies at the host with bundle install
. Build the gem using gem build command
. Install it with gem install to packaged file system
| <mount_point>/bin/<entry_point base name> (i.e., binstub is expected)
| One
| One
| Any
| 5
| Rails project
| Deploy project to packaged filesystem using bundle install
| <mount_point>/local/<entry_point base name>
| No
| One
| Any
| Error | Error: Two or more *.gem files present |
|---|---|
| - | |
| No | |
| No | |
| Two or more |
| Error | Error: Two or more *.gemspec files present |
|---|---|
| - | |
| Two or more | |
| Any | |
| Any |
|===
These scenarios determine how the project is packaged and where the entry point is located within the packaged filesystem.
== Run-time options
Generally Tebako package passes command line options to the packaged application
For example, if the package was created with the following command
running
will be translated by Tebako bootstrap code to
====
However there are several command-line parameters that are intercepted processed by Tebako bootstrap code as follows
=== Image extraction (--tebako-extract option)
Tebako provides an option to an extract its DwarFS filesystem from a package to a local folder for verification or execution.
Where,
<root folder for extracted filesystem>::
The root folder for the extracted filesystem (optional, defaults to source_filesystem)
Extracting Tebako content from the metanorma package:
====
The --tebako-extract option actually runs the following Ruby script:
=== Mounting Host Folder to Tebako Memfs (--tebako-mount option)
Some programs unconditionally use folders located under the application root, and when processed by Tebako or similar tools, these folders are included in the packaging.
For example, there is no configuration option to change where Rails expects the tmp folder to be.
The location is hardcoded in multiple places within the Rails codebase, residing under the application root,
and as a result, it gets included in the read-only Tebako memfs. Although patches have been proposed
(e.g., https://github.com/rails/rails/issues/39583), there is currently no way to change the paths for
temporary files, caches, and sockets.
To address this limitation in Rails and similar issues in other applications, Tebako provides an option to mount a host folder to the memfs tree.
When using Tebako, consider the packaging scenario mentioned above, as it defines the layout of the application
tree. The --tebako-extract option may be useful for understanding the placement of files and folders.
rails.tebako package with $PWD/tmp mounted as local/tmp in the memfs.
Any remaining command-line parameters are passed to the application.
[source,sh]====
--tebako-mount option has the following syntax:
[source,sh]The --tebako-mount option can be repeated multiple times to mount more than one object. The memfs path
is relative to the memfs root, and it is recommended to use absolute paths for host objects. Both directories
and files can be mounted in this way. Tebako allows overlaying existing memfs objects, so there are no significant
limitations.
== Trivia: origin of name
"tamatebako" (玉手箱) is the treasure box given to Urashima Taro in the Ryugu, for which he was asked not to open if he wished to return. He opened the box upon the shock from his return that three hundred years has passed. Apparently what was stored in the box was his age.
This packager was made to store Ruby and its gems, and therefore named after the said treasure box (storing gems inside a treasure box).
Since "tamatebako" is rather long for the non-Japanese speaker, we use "tebako" (手箱, also "tehako") instead, the generic term for a personal box.
== Contributing
We welcome contributions! Please see our contributing guidelines for more information.
== License
Copyright Ribose. All rights reserved.
Tebako is released under the BSD 2-Clause License. See the LICENSE file for details.