search

fullstaq-ruby / server-edition

A server-optimized Ruby distribution: less memory, faster, easy to install and security-patch via APT/YUM
https://fullstaqruby.org
MIT License
607 stars 30 forks source link
hacktoberfest ruby ruby-distribution

readme

Fullstaq Ruby Server Edition: a server-optimized Ruby distribution

Fullstaq Ruby is a Ruby distribution that's optimized for use in servers. It is the easiest way to:

You can think of Fullstaq Ruby as a competitor of apt/yum install ruby, rbenv install and rvm install. We supply native OS packages for various Ruby versions, which are optionally compiled with Jemalloc or malloc_trim, allowing for lower memory usage and potentially increased performance. Our packaging method allows much easier security patching.

Fullstaq Ruby is a work-in-progress! Features like editions optimized for containers and Heroku, better system integration, and much more, are planned. Please see our roadmap.

Table of contents:

Key features

Background

Why was Fullstaq Ruby created?

Fullstaq Ruby came about for two reasons:

Who is behind Fullstaq Ruby?

If you like this work, please star this repo, follow @honglilai on Twitter, and/or contact Fullstaq. Fullstaq can take your technology stack to the next level by providing consultancy, training, and much more.

How it works

See also: Installation

The Fullstaq Ruby native OS packages allow you to install Rubies by adding our repository and installing them through the OS package manager. The highlights are:

Package organization

See also: Installation

Let's say you're on Ubuntu (Enterprise Linux packages use a different naming scheme). Let's pretend Fullstaq Ruby only packages Ruby 3.0.2 and Ruby 3.0.3 (and let's pretend the latter is also the latest release). You will be able to install the following packages:

All these packages can be installed in parallel. None of them conflict with each other, not even the variants.

Rbenv integration

Suppose you install all packages listed in the Package organization example. That will register Rubies in the system-wide Rbenv versions directory:

These registrations are symlinks. The actual Ruby installation is in /usr/lib/fullstaq-ruby. So for example, one symlink looks like this:

/usr/lib/rbenv/versions/3.0 -> /usr/lib/fullstaq-ruby/versions/3.0

If you run rbenv versions, you'll see:

$ rbenv versions
* system (set by /home/hongli/.rbenv/version)
  3.0
  3.0-jemalloc
  3.0-malloctrim
  3.0.2
  3.0.2-jemalloc
  3.0.2-malloctrim
  3.0.3
  3.0.3-jemalloc
  3.0.3-malloctrim

Installed Fullstaq Rubies are available to all users on the system. They complement any Rubies that may be installed in ~/.rbenv/versions.

You activate a specific version by using regular Rbenv commands:

 $ rbenv local 3.0.3
 $ rbenv exec ruby -v
 ruby 3.0.3

 $ rbenv local 3.0.3-jemalloc
 $ rbenv exec ruby -v
 ruby 3.0.3 (jemalloc variant)

Minor version packages: a great way to keep Ruby security-patched

fullstaq-ruby-3.0.2 and fullstaq-ruby-3.0.3 are tiny version packages. They package a specific tiny version.

fullstaq-ruby-3.0 is a minor version package. It always contains the latest tiny version! If today the latest Ruby 3.0 version is 3.0.3, then fullstaq-ruby-3.0 contains 3.0.3. If tomorrow 3.0.4 is released, then fullstaq-ruby-3.0 will be updated to contain 3.0.4 instead.

We recommend installing the minor version package/image over installing tiny version packages/images:

Tip: You should also configure automatic updates (unattended upgrades).

About variants

So there are 3 variants: normal, jemalloc, malloctrim. What are the differences?

Recommendation: use the jemalloc variant, unless you actually observe compatibility problems.

Comparisons to other systems

Vs RVM and Rbenv

RVM and Rbenv are Ruby version managers. They allow you to install and switch between multiple Ruby versions. They allow installing Ruby by compiling from source, or sometimes by downloading precompiled binaries (the availability of binaries for different platforms is limited).

Fullstaq Ruby is not a Ruby version manager. Fullstaq Ruby is a Ruby distribution: we supply binaries for Ruby. You can think of Fullstaq Ruby as a replacement for rvm install and rbenv install.

The differences between rvm/rbenv install and Fullstaq Ruby are:

Vs Ruby packages included in operating systems' official repositories

Vs the Brightbox PPA

The Brightbox PPA is an Ubuntu APT repository provided by Brightbox.

Like Fullstaq Ruby, the Brightbox PPA contains packages for multiple Ruby versions, for multiple Ubuntu versions.

The differences are:

Vs JRuby, TruffleRuby and Rubinius

JRuby, TruffleRuby and Rubinius are alternative Ruby implementations, that are different from the official Ruby (MRI).

Fullstaq Ruby is not an alternative Ruby implementation. It is a distribution of MRI.

Vs LD_PRELOADing Jemalloc yourself

You can enjoy reduced memory usage and higher performance with a do-it-yourself solution that involves LD_PRELOADing Jemalloc. The difference with Fullstaq Ruby is not on a technical level, but on a service level.

With a DIY solution, you are responsible for lifecycle management and for ensuring that the right version of Jemalloc is picked. Only Jemalloc 3 yields reduced memory usage, Jemalloc 5 does not. If you for example install Jemalloc from your distribution's package manager, then you must double-check that your distribution doesn't distribute a too new Jemalloc.

You can of course install Jemalloc from source (assuming you don't mind compiling). But then you become responsible for keeping it security-patched.

Fullstaq Ruby, ensures that the right version of Jemalloc is used. We are a bunch of people that care about this subject, so we are constantly researching better ways to integrate Jemalloc. For example there are some efforts on the way to research how to make use of Jemalloc 5. We also keep an eye on security issues and supply security updates, so that you can sit back and relax.

Installation

Enterprise Linux

Red Hat Enterprise Linux (RHEL) is the original "Enterprise Linux". Compatible derivatives are CentOS, Rocky Linux and Alma Linux.

Add the Fullstaq Ruby repository by creating /etc/yum.repos.d/fullstaq-ruby.repo. Pick one of the following:

Enterprise Linux 9:

[fullstaq-ruby]
name=fullstaq-ruby
baseurl=https://yum.fullstaqruby.org/el-9/$basearch
gpgcheck=0
repo_gpgcheck=1
enabled=1
gpgkey=https://raw.githubusercontent.com/fullstaq-ruby/server-edition/main/fullstaq-ruby.asc
sslverify=1

Enterprise Linux 8:

[fullstaq-ruby]
name=fullstaq-ruby
baseurl=https://yum.fullstaqruby.org/centos-8/$basearch
gpgcheck=0
repo_gpgcheck=1
enabled=1
gpgkey=https://raw.githubusercontent.com/fullstaq-ruby/server-edition/main/fullstaq-ruby.asc
sslverify=1

Then install fullstaq-ruby-common:

sudo yum install fullstaq-ruby-common

Ruby packages are now available as fullstaq-ruby-<VERSION>:

$ sudo yum search fullstaq-ruby
...
fullstaq-ruby-3.0.x86_64 : Fullstaq Ruby 3.0
fullstaq-ruby-3.0-jemalloc.x86_64 : Fullstaq Ruby 3.0-jemalloc
fullstaq-ruby-3.0-malloctrim.x86_64 : Fullstaq Ruby 3.0-malloctrim
...
fullstaq-ruby-3.1.x86_64 : Fullstaq Ruby 3.1
fullstaq-ruby-3.1-jemalloc.x86_64 : Fullstaq Ruby 3.1-jemalloc
fullstaq-ruby-3.1-malloctrim.x86_64 : Fullstaq Ruby 3.1-malloctrim
...

You can either install a specific tiny version....

sudo yum install fullstaq-ruby-3.0.3

...or (recommended!) you can install the latest tiny version of a minor release (e.g. the latest Ruby 3.1):

# This will auto-update to the latest tiny version when it's released
sudo yum install fullstaq-ruby-3.1

You can even install multiple versions in parallel if you really want to:

# Installs the latest 3.1
sudo yum install fullstaq-ruby-3.1

# In parallel, *also* install Ruby 3.1.1
sudo yum install fullstaq-ruby-3.1.1

Next steps:

Debian/Ubuntu

First, make sure your package manager supports HTTPS and that the necessary crypto tools are installed:

sudo apt install gnupg apt-transport-https ca-certificates curl

Next, add the Fullstaq Ruby repository by creating /etc/apt/sources.list.d/fullstaq-ruby.list. Paste one of the lines below depending your distro:

# Ubuntu 24.04
deb https://apt.fullstaqruby.org ubuntu-24.04 main

# Ubuntu 22.04
deb https://apt.fullstaqruby.org ubuntu-22.04 main

# Ubuntu 20.04
deb https://apt.fullstaqruby.org ubuntu-20.04 main

# Debian 12
deb [Signed-By=/etc/apt/fullstaq-ruby.asc] https://apt.fullstaqruby.org debian-12 main

# Debian 11
deb https://apt.fullstaqruby.org debian-11 main

# Debian 10
deb https://apt.fullstaqruby.org debian-10 main

Then download the Fullstaq Ruby public key and update APT:

# On Debian >= 12 and Ubuntu >= 24.04:
sudo curl -SLfo /etc/apt/fullstaq-ruby.asc https://raw.githubusercontent.com/fullstaq-ruby/server-edition/main/fullstaq-ruby.asc
sudo apt update

# Otherwise:
sudo curl -SLfo /etc/apt/trusted.gpg.d/fullstaq-ruby.asc https://raw.githubusercontent.com/fullstaq-ruby/server-edition/main/fullstaq-ruby.asc
sudo apt update

Then install fullstaq-ruby-common:

sudo apt install fullstaq-ruby-common

Ruby packages are now available as fullstaq-ruby-<VERSION>:

$ sudo apt search fullstaq-ruby
...
fullstaq-ruby-3.0/ubuntu-22.04 1-ubuntu-22.04 amd64
  Fullstaq Ruby 3.0

fullstaq-ruby-3.0-jemalloc/ubuntu-22.04 1-ubuntu-22.04 amd64
  Fullstaq Ruby 3.0-jemalloc

fullstaq-ruby-3.0-malloctrim/ubuntu-22.04 1-ubuntu-22.04 amd64
  Fullstaq Ruby 3.0-malloctrim
...
fullstaq-ruby-3.1/ubuntu-22.04 1-ubuntu-22.04 amd64
  Fullstaq Ruby 3.1

fullstaq-ruby-3.1-jemalloc/ubuntu-22.04 1-ubuntu-22.04 amd64
  Fullstaq Ruby 3.1-jemalloc

fullstaq-ruby-3.1-malloctrim/ubuntu-22.04 1-ubuntu-22.04 amd64
  Fullstaq Ruby 3.1-malloctrim
...

You can either install a specific tiny version....

sudo apt install fullstaq-ruby-3.1.1

...or (recommended!) you can install the latest tiny version of a minor release (e.g. the latest Ruby 3.1):

# This will auto-update to the latest tiny version when it's released
sudo apt install fullstaq-ruby-3.1

You can even install multiple versions in parallel if you really want to:

# Installs the latest 3.1
sudo apt install fullstaq-ruby-3.1

# In parallel, *also* install Ruby 3.1.1
sudo apt install fullstaq-ruby-3.1.1

Next steps:

Deactivate Git-based Rbenv

Note: you only need to perform this step if you already had a Git-based Rbenv installed. Otherwise you can skip to the next step: Activate Rbenv shell integration.

Fullstaq-Ruby relies on a sightly modified version of Rbenv, for which we supply an OS package. This package installs the rbenv binary to /usr/bin.

You should modify your shell files to remove your Git-based Rbenv installation from your PATH, so that /usr/bin/rbenv is used instead. For example in your .bash_profile and/or .bashrc, remove lines that look like this:

# REMOVE lines like this!
export PATH="$HOME/.rbenv/bin:$PATH"

There is no need to remove eval "$(rbenv init -)". You're still going to use Rbenv — just one that Fullstaq Ruby provides. More about that in the next step, Activate Rbenv shell integration.

There is also no need to remove the ~/.rbenv directory. The Ruby versions installed in there are still supported — in addition to system-wide ones installed by the Fullstaq Ruby packages.

Activate Rbenv shell integration (optional)

Note: you only need to perform this step if you didn't already have Rbenv shell integration installed. You can skip this step if you already had Rbenv shell integration installed from a previous Git-based Rbenv installation.

For an optimal Rbenv experience, you should activate its shell integration.

Run this command, which will tell you to add some code to one of your shell files (like .bashrc or .bash_profile):

/usr/bin/rbenv init

Be sure to restart your shell after installing shell integration.

System-wide shell integration

Adding to .bashrc/.bash_profile only activates the shell integration for that specific user. If you want to activate shell integration for all users, you should add to a system-wide shell file. For example if you're using bash, then:

Usage after installation

Using a specific Ruby version

Ruby versions are installed to /usr/lib/fullstaq-ruby/versions/<VERSION>. Each such directory has a bin subdirectory which contains ruby, irb, gem, etc.

Suppose you installed Ruby 3.3 (normal variant). You can execute it directly:

$ /usr/lib/fullstaq-ruby/versions/3.3/bin/ruby --version
ruby 3.3.0

$ /usr/lib/fullstaq-ruby/versions/3.3/bin/gem install --no-document nokogiri
...

But for convenience, it's better to add /usr/lib/fullstaq-ruby/versions/<VERSION>/bin to your PATH so that you don't have to specify the full path every time. See Activate Rbenv shell integration (optional).

Usage with Rbenv

The recommended way to use Fullstaq Ruby is through the Rbenv integration. You should learn Rbenv if you are not familiar with it. Here is a handy cheat sheet.

Suppose you installed Ruby 3.3 (normal variant). You can run that Ruby by setting RBENV_VERSION and prefixing yours commands with rbenv exec:

$ export RBENV_VERSION=3.3
$ rbenv exec ruby --version
ruby 3.3.0
$ rbenv exec gem env
...

Or, if you've activated the Rbenv shell integration, just running ruby, gem and various other Ruby would also work, provided that you've activated a certain version and that there's an Rbenv shim available:

$ rbenv local 3.3
$ ruby --version
ruby 3.3.0
$ gem env
...

Installing gems and root privileges

By default, gems are installed into a system location (/usr/lib/fullstaq-ruby/versions/XXX/lib/ruby/gems/XXX). This means:

You can also choose to install gems into a user's home directory, or some other directory that does not require root privileges.

Installing gems system-wide with sudo

Be sure to run gem install with sudo:

# When not using Rbenv
sudo /usr/lib/fullstaq-ruby/versions/XXX/bin/gem install GEM_NAME_HERE

# When using Rbenv
sudo env RBENV_VERSION=XXX rbenv exec gem install GEM_NAME_HERE

(replace XXX with the desired Ruby version and variant for which you want to install the gem)

When using Bundler, do not prepend sudo. Bundler will run sudo for you automatically.

Installing gems without root privileges

Be sure to run gem install with --user-install, which will install gems to the user's home directory:

# When not using Rbenv
/usr/lib/fullstaq-ruby/versions/XXX/bin/gem install GEM_NAME_HERE --user-install

# When using Rbenv (assuming you have the shell integration enabled)
gem install GEM_NAME_HERE --user-install

(replace XXX with the desired Ruby version and variant for which you want to install the gem)

When using Bundler, pass --path and point it to a user-writable location:

bundle install --path vendor/bundle

Passenger for Nginx/Apache integration

First, find out the full path to the Ruby version's binary that you want to use: Using a specific Ruby version.

Next, specify it in the Nginx or Apache config file:

Restart Nginx or Apache after you've made the change.

Puma, Unicorn or Passenger Standalone integration

First, find out the full path to the Ruby version's binary that you want to use: Using a specific Ruby version.

Next, modify your Puma/Unicorn/Passenger Standalone startup script (e.g. Systemd unit) and make sure that it executes your application server using the full path to your Ruby executable. This is usually done by modifying the ExecStart option.

For example suppose you have a Systemd unit file /etc/systemd/system/puma.service that looks like this:

# Just an example!

[Unit]
Description=Puma HTTP Server
After=network.target

[Service]
Type=simple
User=app
WorkingDirectory=<YOUR_APP_PATH>
ExecStart=/home/app/.rbenv/versions/3.3.0/bin/ruby -S bundle exec puma -C puma.rb
Restart=always

[Install]
WantedBy=multi-user.target

Make sure that your ExecStart command is prefixed by a call to /full-path-to-ruby -S, like this:

ExecStart=/usr/lib/fullstaq-ruby/versions/3.3.0-jemalloc/bin/ruby -S bundle exec puma -C puma.rb

Don't forget the -S!

Restart your application server after you've made a change, for example sudo systemctl restart puma.

Capistrano integration

If you use Capistrano to deploy your app, then you should use the capistrano-rbenv plugin. Requirements:

In your deploy/config.rb make sure you set rbenv_type to :fullstaq, and rbenv_ruby to the Ruby version to you want, possibly with a variant suffix. Examples:

set :rbenv_type, :fullstaq

# Use Ruby 3.3 (latest tiny version), normal variant
set :rbenv_ruby, '3.3'

# Use Ruby 3.3.1, normal variant
set :rbenv_ruby, '3.3.1'

# Use Ruby 3.3.1, jemalloc variant
set :rbenv_ruby, '3.3.1-jemalloc'

Automatic updates (unattended upgrades)

On Debian/Ubuntu, you can configure automatic updates using unattended-upgrades. This is especially useful in combination with minor version packages so that you can automatically upgrade Ruby from, say, 3.2.1 to 3.2.2 (but not 3.3).

Make sure unattended-upgrades are installed:

sudo apt install unattended-upgrades

Then in /etc/apt/apt.conf.d/50unattended-upgrades, add Fullstaq-Ruby:<DISTRO AND VERSION> to Unattended-Upgrade::Origins-Pattern. DISTRO AND VERSION should equal one of the APT repository suite names listed in the Debian/Ubuntu installation instructions.

Example:

Unattended-Upgrade::Origins-Pattern {
    ...
    "Fullstaq-Ruby:ubuntu-22.04";
}

FAQ

What is Jemalloc and how does it benefit me?

Jemalloc is the FreeBSD libc memory allocator. It uses a more complex algorithm than the default Linux glibc memory allocator, but is faster and results in less memory fragmentation (which reduces memory usage significantly). Jemalloc has been used successfully by e.g. Firefox and Redis to reduce their memory footprint and ongoing development is supported by Facebook.

What is malloc_trim and how does it benefit me?

malloc_trim() is an API that is part of the glibc memory allocator. In Hongli Lai's research project What Causes Ruby Memory Bloat?, Hongli has identified the OS memory allocator as a major cause of memory bloating in Ruby. Luckily, simple fixes exist, and one fix is to invoke malloc_trim() which tells the glibc memory allocator to release free memory back to the kernel.

It is found that, if Ruby calls malloc_trim() during a garbage collection, then memory usage can be reduced significantly.

However, malloc_trim may have some performance impact.

Is Fullstaq Ruby faster than regular Ruby (MRI)?

If you pick the Jemalloc variant, then yes it is often faster. See About variants. See also these benchmarks:

Why does Fullstaq Ruby integrate with Rbenv?

Many users have a need to install and switch between multiple Rubies on a single machine, so we needed a Ruby version manager. Rather than inventing our own, we chose to use Rbenv because it's popular and because it's easy to integrate with.

I do not need multiple Rubies (and have no need for Rbenv), is Fullstaq Ruby suitable for me?

Yes. The multi-Ruby-support via Rbenv is quite lightweight and is unintrusive, weighting a couple hundred KB at most. Even if you do not need Rbenv, the fact that Fullstaq Ruby uses Rbenv doesn't get in your way and does not meaningfully increase resource utilization.

Which variant should I pick?

See: About variants.

Why a new distribution? Why not contribute to Ruby core?

Main article: What is Fullstaq Ruby's signifiance to the community, and its long-term project vision?

The Ruby core team is reluctant or slow to incorporate certain changes. And for a good reason: whether a change is an improvement depends on the perspective. The Ruby core team has to care about a wide range of users and use cases. Incorporating Jemalloc or malloc_trim is not necessarily an improvement for all their users.

While understandable, that attitude does not help users for which those changes are actual improvement. Fullstaq Ruby's goal is to help people who use Ruby in a server context, in production environments, on x86-64 Linux. For example we don't care about development environments like macOS, or Raspberry PIs. This allows us to make less conservative choices than the Ruby core team.

Furthermore, the Ruby core team does not want to be responsible for certain aspects, such as distributing binaries. But binaries are valuable. So we take up the responsibility of packaging binaries.

The Ruby core team have debated for years on whether to incorporate Jemalloc, and so far they've only been reluctant. Furthermore, Hongli Lai's research and discussions with various experts have revealed that the only way to make optimal use of Jemalloc is through the LD_PRELOAD mechanism: compiling Ruby with --with-jemalloc is not enough! LD_PRELOAD is such an intrusive and platform-specific change, that we're confident that the Ruby core team will never accept using such a mechanism by default.

In short: Fullstaq Ruby's goal is to bring value to server-production users as soon as possible, and we think maintaining our own distribution is the fastest and best way to achieve that goal.

Will Fullstaq Ruby become paid in the future?

Main article: What is Fullstaq Ruby's signifiance to the community, and its long-term project vision?

There will be no paid version. Fullstaq Ruby is fully open source. It is also intended to be a community project where anyone can contribute. There are no monetization plans.

I am wary of vendor lock-in or that I will become dependent on a specific party for supplying packages. What is Fullstaq Ruby's take on this?

Main article: What is Fullstaq Ruby's signifiance to the community, and its long-term project vision?

Vendor lock-in is a valid concern that many of us have. It is something we thought about from the beginning and that we are addressing.

We have architected our systems in such a way that anyone will be able to build packages themselves. If we ever become slow or defunct, then anyone can easily take matters into own hands. Building Fullstaq Ruby packages from your own systems is so simple that nearly anyone can do it: just install Docker, edit a config file and run a command.

This is achieved by automating the entire build process, and by releasing the making the build tooling as open source. There are almost no manual processes. Please take a look at this repository's source code.

Because Fullstaq Ruby is still a work-in-progress, we don't have documentation yet on how to build packages yourself. Such documentation is planned for epic 5 of the roadmap.

Contributing

If you're interested in contributing to Fullstaq Ruby, please check out our contribution guide to get started.

Community — getting help, reporting issues, proposing ideas

To engage with our community, please visit: