Change the use of gzip to another program like pigz or tar

[0xCiBeR]

Describe the feature and why you would like to see it.

Currently when you move a server between nodes, gzip is invoked to create a compressed file and then move that file. The problem with this is gzip is single threaded; this makes it take a lot of time for big sized servers 10+ GB

The feature request is to implement something like http://zlib.net/pigz/ or just use a faster non comprssion method like tar https://www.gnu.org/software/tar/

Moving big files across dedicated machines shouldn't be a problem, so i find that big files(aka not compressed) could also work in this scenario

Describe the solution you'd like.

Change program used to compress server backups when moving servers between nodes.

Optional additional context to this request.

No response

[Lunaversitay]

This isn't really much of an issue with the compression library as it's more of an issue with how NodeJS handles child processes. Unfortunately swapping the method won't do much however in the daemon upgrade we do have a better solution for server transfers we want to utilize over compressing -> moving to new node -> decompressing.

Going to go ahead and close this since the method will likely not be changed.

[0xCiBeR]

Hello @Lunaversitay can you please tell me how you reached such conclusion? Is there any benchmarks or test ran that can be checked out? I think this is ignoring the elephant in the room, servers take way to much to compress as its being run single threaded.

[0xCiBeR]

Here are some actual benchmarks over the same heavy files:

PIGZ:

real    0m51.001s
user    21m22.344s
sys     1m10.660s

GZIP:

real    5m18.999s
user    5m0.224s
sys     0m18.650s

Both tests have been run on a AMD 7950x with 192gb of ram.

Here is another test using NodeJS childs and tar previously:

node test
Compressing with gzip...
gzip compression took 49.152975929 seconds.
Compressing with pigz...
pigz compression took 30.189148816 seconds.

Script:

const { exec } = require('child_process');
const { performance } = require('perf_hooks');

const directory = 'test_dir';
const gzipOutput = 'gzip_output.tar.gz';
const pigzOutput = 'pigz_output.tar.gz';

// Function to execute a command and measure its time
const measureTime = (command, outputFile) => {
  return new Promise((resolve, reject) => {
    const startTime = performance.now();

    exec(command, (error, stdout, stderr) => {
      const endTime = performance.now();
      const duration = endTime - startTime;

      if (error) {
        reject(`Error: ${error.message}`);
      } else if (stderr) {
        console.error(`Stderr: ${stderr}`); // Log stderr to console
      }

      resolve(duration);
    });
  });
};

const runComparisons = async () => {
  try {
    console.log('Compressing with gzip...');
    const gzipTime = await measureTime(
      `tar -cf - ${directory} | gzip > ${gzipOutput}`,
      gzipOutput
    );
    console.log(`gzip compression took ${gzipTime / 1000} seconds.`);

    console.log('Compressing with pigz...');
    const pigzTime = await measureTime(
      `tar -cf - ${directory} | pigz > ${pigzOutput}`,
      pigzOutput
    );
    console.log(`pigz compression took ${pigzTime / 1000} seconds.`);
  } catch (error) {
    console.error(error);
  }
};

runComparisons();

[Lunaversitay]

Apologies my initial response didn't mention this but this issue is being tracked internally with the go daemon resuming development. While we saw slight improvements swapping methods it wasn't anything significant, viction mentioned he'll look more into it in the discord when he has the free time. But for the most part we will see these performance issues with compressing / server moving / decompressing completely dissolved when the go daemon is released :)