Exit Ruby process to release memory resource

[supermomonga]

Background

Currently, this gem launches Litestream as a subprocess, and the original Ruby process waits for the termination of the subprocess.

I am using this gem in a Rails project. Since Rails projects generally use about 200 to 300MB of RAM, keeping the Ruby process alive means continuously consuming server's memory resources unnecessarily.

Proposal

Terminate the Ruby process and keep only the Litestream process alive.

Implementation

Use fork and exec instead of system.

• Original process: continue rest of program after fork. (currently, just exit immediately after fork.)
• Forked process: call exec and replace itself with new Litestream process

Result

On my system, I was able to reduce the memory usage from approximately 320MB to about 30MB.

[fractaledmind]

I like the Result here, but before I merge I need to ensure that I 100% understand what this code is doing, and I confess that I don't yet. Can you walk me through what is happening in more detail so that I can get on the same page as you. I will definitely merge this, just need to fully download the context and relevant information first.

[supermomonga]

@fractaledmind

Of course! But I'm not familiar with English, so let me explain using Ruby code as an example.

The main purpose is to replace system with exec. https://rubydoc.info/stdlib/core/Kernel:exec

A simple example:

system version

puts 'before exec'
system('sleep 300')
# The Ruby process waits for the "sleep" command to finish, then the line below will be processed after 300 seconds.
puts 'after exec'

# stdout:
# => before exec
# => after exec

exec version

puts 'before exec'
exec('sleep 300')
# The Ruby process will exit when `exec` is called, so the line below will not be processed.
puts 'after exec'

# stdout:
# => before exec

In the exec version, the original process (Ruby) will be immediately terminated when the exec method is called, so the remaining code will not be executed.

This is useful for releasing machine resources without waiting for the launched process (in this case, sleep) to finish.

---

# Ruby process consumes 1GB of RAM
large_object = get_1gigabytes_object()

# Execute `sleep` and exit without waiting 300 seconds
exec('sleep 300')

In this case, if I use system, the Ruby process consuming 1GB of RAM will stay alive for about 300 seconds.

---

The problem is, if I call exec, the original process will terminate immediately. Sometimes I want to do something after launching the command.

exec('sleep 300')
puts "Sleep started!"

In this example, puts will not be executed.

Then, I'll use fork. Fork creates a new process, and the return value is different between the original and the new processes.

forked_process_pid = fork()
if forked_process_pid.nil?
  puts "I'm the forked process. My PID is #{ Process.pid }."
else
  puts "I'm the original process. My PID is #{ Process.pid }, and the forked process's PID is #{forked_process_pid}."
end

# output:
# => I'm the original process. My PID is 1775230 and the forked process's PID is 1775338.
# => I'm the forked process. My PID is 1775338.

---

So, if I use fork and exec, I can manage to gracefully finish the entire Ruby script (without forced termination) and don't need to wait for a long-running command to finish.

if fork.nil?
  puts "I'm the forked process. Launching litestream and replacing myself with the litestream process."
  exec(*command)
end

# A forked Ruby process never reaches here. Only original process does.
puts "I'm the original process. Gracefully exiting..."
some_method_before_exit()

---

Did I explain it well? I'm not very confident...

[fractaledmind]

That was very clear, yes; thank you. I am waiting to merge until I think thru how I want to handle replication more generally tho. Specifically, I am thinking that the gem should push devs to wrap their server process with litestream replicate -exec instead of running replication in a separate process.

[fractaledmind]

Released with version 0.3.3