Cross-platform backup & restore

[mtruyens]

I'm trying to use backups & restores across platforms, e.g. from a Linux server in production to a dev environment on a Mac.

Obviously, the paths and config files differ. However, no matter what I try, I always get errors during restore — e.g.

[Error] Cannot write to backup directory - "/"

Any possibility to outline the steps, or things to watch out for? The documentation is silent on this point :-(

Note that my backups are compressed, but this should not be an issue.

[donhardman]

Hello,

Could you please provide additional details, such as the configuration of the searchd, the exact parameters for your backup script, and the exact parameters you use for restoration?

In instances where absolute paths are used, the process should be identical in OSX. Have you attempted to create a backup using a relative data directory configuration? This approach would allow the root path to vary between different operating systems.

[mtruyens]

I believe the most important different consists of the location of the standard folders, which is quite different between a standard Linux install and a standard Homebrew installation on a mac.

Here's the configuration at the Linux server:

common {
    plugin_dir = /usr/local/lib/manticore
}

searchd {
    listen = 127.0.0.1:9308:http_readonly
    listen = 127.0.0.1:9309:http
    qcache_thresh_msec = 0
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    query_log_format = sphinxql
}

indexer {
       mem_limit = 1G
}

Here's the configuration for the local development (Mac) machine:

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    log = /opt/homebrew/var/log/manticore/searchd.log
    query_log = /opt/homebrew/var/log/manticore/query.log
    pid_file = /opt/homebrew/var/run/manticore/searchd.pid
    data_dir = /opt/homebrew/var/manticore
    query_log_format = sphinxql
}

The backup command on the Linux server is

manticore-backup --compress --backup-dir=$BACKUP_DIR --config=/etc/manticoresearch/manticore.conf

The restore command I tried on my Mac is:

sudo manticore-backup --backup-dir=/Users/mtruyens/Data/home/mtruyens/backups --restore=backup-20230826030001

Which results in :

Manticore config file: 
Backup dir: /Users/mtruyens/Data/home/mtruyens/backups
2023-08-31 07:37:37 [Info] Starting to restore...

Manticore config
  endpoint =  http://127.0.0.1:9309
2023-08-31 07:37:38 [Info] Restoring config files...
2023-08-31 07:37:38 [Error] Cannot write to backup directory - "/"

[donhardman]

Hi there.

Our team has reviewed the issue and concluded that it's a rather unique situation. It may be challenging to implement the ability to auto-detect prefixes in various OSes. We're currently maintaining absolute prefixes for each folder in the backup structure. Even if we were to introduce a parameter that allows setting a prefix for restoration, we still have different default prefixes for other directories. Sometimes, these differ across platforms, such as the path to the data directory in Linux and OSX.

As a potential solution, try a Dockerized environment with Linux. If you need Manticore on your OSX, the simplest way is to run a Linux container, restore it in the same environment, and then use it on your host machine by employing a network mode host.

We also offer support for mysqldump. If that suits your needs, it can be used to restore your backup in a cross-platform environment. For more details, please visit: https://manual.manticoresearch.com/Securing_and_compacting_a_table/Backup_and_restore#Backup-and-restore-with-mysqldump

Remember, you cannot restore a full backup with a running daemon. In the case of a full backup, the daemon should not be operational on the system. Also, you don't need to create a config file.

[sanikolaev]

As a potential solution, try a Dockerized environment with Linux

Or you can write a simple bash script which will restore from the backup by copying the files to proper places and uncompressing. manticore-backup doesn't do anything special in this case (unlike the backup phase when it has to freeze tables).