search

manticoresoftware / manticoresearch-backup

Repository for Manticore Search backup scripts
GNU General Public License v3.0
3 stars 1 forks source link

Error backup restoring #88

Closed djklim87 closed 10 months ago

djklim87 commented 10 months ago

When I try to restore backup (attached to this issue) I get error searchd error

2023-10-31T13:54:13.008952876Z WARNING: table 'v': disabled at the JSON config
2023-10-31T13:54:13.008963289Z WARNING: table 'v': prealloc: failed to open /var/lib/manticore/v/v.lock: No such file or directory - NOT SERVING

Restore command:

manticore-backup --force --backup-dir='/' --restore='docker-entrypoint-initdb.d'

It's because manticore-backup wrong substituted indexes path.

In original backup indexes are stored in /var/lib/manticore/v folder but after restore these files are located directly at /var/lib/manticore:

root@1aef91416b53:/var/lib/manticore# ls -la
total 48
drwxr-xr-x 1 manticore manticore 4096 Oct 31 13:54 .
drwxr-xr-x 1 root      root      4096 Oct 31 13:53 ..
drwxr-xr-x 3 manticore manticore 4096 Oct 31 13:54 .extra
-rw-r--r-- 1 manticore manticore    8 Oct 31 13:54 .extra.installed
drwxr-xr-x 2 manticore manticore 4096 Oct 31 13:54 .mcl
drwx------ 2 manticore manticore 4096 Oct 31 13:54 binlog
-rw-r--r-- 1 manticore manticore   84 Oct 31 13:54 manticore.json
drwxr-xr-x 2 manticore manticore 4096 Oct 31 13:53 replication
-rw------- 1 manticore manticore    0 Oct 31 13:54 state.sql
-rw-r--r-- 1 manticore manticore  503 Oct 31 13:54 v.meta
-rw-r--r-- 1 manticore manticore  126 Oct 31 13:54 v.ram
-rw-r--r-- 1 manticore manticore    3 Oct 31 13:54 v.settings
djklim87 commented 10 months ago

backup-20231031132826.zip

sanikolaev commented 10 months ago

Please provide a better reproducible case as it works fine for me with the provided backup in Ubuntu Jammy (from https://github.com/manticoresoftware/linux_dockers)

root@0d71deb22c5c:~# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy

root@0d71deb22c5c:~# wget https://github.com/manticoresoftware/manticoresearch-backup/files/13217281/backup-20231031132826.zip

root@0d71deb22c5c:~# unzip backup-20231031132826.zip

root@0d71deb22c5c:~# manticore-backup --force --backup-dir='/root/' --restore='backup-20231031132826'
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore config file:
Backup dir: /root/
2023-10-31 15:26:56 [Info] Stored versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-10-31 15:26:56 [Info] Current versions: {"manticore":"6.2.13 346801349@231030 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-10-31 15:26:56 [Info] Warning: You try to restore backup of the different version. Use --force to bypass this.
2023-10-31 15:26:56 [Info] Starting to restore...

Manticore config
  endpoint =  http://172.23.0.4:9312
2023-10-31 15:26:56 [Info] Restoring config files...
2023-10-31 15:26:56 [Info]   config files - OK
2023-10-31 15:26:56 [Info] Restoring state files...
2023-10-31 15:26:56 [Info]   state files - OK
2023-10-31 15:26:56 [Info] Restoring data files...
2023-10-31 15:26:56 [Info]   tables' files - OK
2023-10-31 15:26:56 [Info] The backup '/root/backup-20231031132826' was successfully restored.
2023-10-31 15:26:56 [Info] Elapsed time: 0.05s
2023-10-31 15:26:56 [Info] Done

The manticore-backup version was:

root@0d71deb22c5c:~# manticore-backup --version
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore Backup version: 1.1.1
djklim87 commented 10 months ago

No, issue still reproducable:

apt-get update && apt-get install -y wget gnupg unzip && \
wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && sudo dpkg -i manticore-dev-repo.noarch.deb && sudo apt -y update && sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l && \
wget https://github.com/manticoresoftware/manticoresearch-backup/files/13217281/backup-20231031132826.zip && unzip backup-20231031132826.zip && \
manticore-backup --version && \
find /backup-20231031132826/config -type f -exec sh -c 'rm -f "${1#/backup-20231031132826/config}"' sh {} \; &&\
find /backup-20231031132826/state -type f -exec sh -c 'rm -f "${1#/backup-20231031132826/state}"' sh {} \; &&\
manticore-backup --force --backup-dir=/ --restore=backup-20231031132826

Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore Backup version: 1.1.1
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore config file: 
Backup dir: /
2023-11-01 12:09:04 [Info] Stored versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-01 12:09:04 [Info] Current versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-01 12:09:04 [Info] Starting to restore...

Manticore config
  endpoint =  http://172.21.0.5:9312
2023-11-01 12:09:04 [Info] Restoring config files...
2023-11-01 12:09:04 [Info]   config files - OK
2023-11-01 12:09:04 [Info] Restoring state files...
2023-11-01 12:09:04 [Info]   state files - OK
2023-11-01 12:09:04 [Info] Restoring data files...
2023-11-01 12:09:04 [Info]   tables' files - OK
2023-11-01 12:09:04 [Info] The backup '//backup-20231031132826' was successfully restored.
2023-11-01 12:09:04 [Info] Elapsed time: 0.03s
2023-11-01 12:09:06 [Info] Done

Next we see that files not in desired folder (/var/lib/manticore/v), instead it restored to /var/lib/manticore/

root@82278976cb76:/# ls /var/lib/manticore/
manticore.json  v.meta  v.ram  v.settings
sanikolaev commented 10 months ago

I confirm it loses the v dir in this case.

donhardman commented 10 months ago

Cannot reproduce it on the latest version, please try again with 1.1.3

Image

sanikolaev commented 10 months ago

Can't reproduce in Ubuntu Jammy anymore:

root@755ab9783c2d:/# apt-get update && apt-get install -y wget gnupg unzip && wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && sudo dpkg -i manticore-dev-repo.noarch.deb && sudo apt -y update && sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l && wget https://github.com/manticoresoftware/manticoresearch-backup/files/13217281/backup-20231031132826.zip && unzip backup-20231031132826.zip && manticore-backup --version && find /backup-20231031132826/config -type f -exec sh -c 'rm -f "${1#/backup-20231031132826/config}"' sh {} \; &&find /backup-20231031132826/state -type f -exec sh -c 'rm -f "${1#/backup-20231031132826/state}"' sh {} \; &&manticore-backup --force --backup-dir=/ --restore=backup-20231031132826

...

2023-11-02 09:06:45 (91.1 MB/s) - 'backup-20231031132826.zip' saved [5553/5553]

Archive:  backup-20231031132826.zip
   creating: backup-20231031132826/
   creating: backup-20231031132826/config/
   creating: backup-20231031132826/config/etc/
   creating: backup-20231031132826/config/etc/manticoresearch/
  inflating: backup-20231031132826/config/etc/manticoresearch/manticore.conf
   creating: backup-20231031132826/config/var/
   creating: backup-20231031132826/config/var/lib/
   creating: backup-20231031132826/config/var/lib/manticore/
  inflating: backup-20231031132826/config/var/lib/manticore/manticore.json
   creating: backup-20231031132826/data/
   creating: backup-20231031132826/data/v/
  inflating: backup-20231031132826/data/v/v.meta
  inflating: backup-20231031132826/data/v/v.ram
  inflating: backup-20231031132826/data/v/v.settings
   creating: backup-20231031132826/state/
  inflating: backup-20231031132826/versions.json
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore Backup version: 1.1.3
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore config file:
Backup dir: /
2023-11-02 09:06:45 [Info] Stored versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-02 09:06:45 [Info] Current versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-02 09:06:45 [Info] Starting to restore...

Manticore config
  endpoint =  http://172.23.0.15:9312
2023-11-02 09:06:45 [Info] Restoring config files...
2023-11-02 09:06:45 [Info]   config files - OK
2023-11-02 09:06:45 [Info] Restoring state files...
2023-11-02 09:06:45 [Info]   state files - OK
2023-11-02 09:06:45 [Info] Restoring data files...
2023-11-02 09:06:45 [Info]   tables' files - OK
2023-11-02 09:06:45 [Info] The backup '//backup-20231031132826' was successfully restored.
2023-11-02 09:06:45 [Info] Elapsed time: 0.02s
2023-11-02 09:06:45 [Info] Done

The version is:

root@755ab9783c2d:/# manticore-backup --version
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore Backup version: 1.1.3

Not sure how I reproduced it before.

djklim87 commented 10 months ago

No the problem still available. Backup restores successfully but it still put table into wrong dir:

apt-get update && apt-get install -y wget gnupg unzip && wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && sudo dpkg -i manticore-dev-repo.noarch.deb && sudo apt -y update && sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l && wget https://github.com/manticoresoftware/manticoresearch-backup/files/13217281/backup-20231031132826.zip && unzip backup-20231031132826.zip && manticore-backup --version && find /backup-20231031132826/config -type f -exec sh -c 'rm -f "${1#/backup-20231031132826/config}"' sh {} \; &&find /backup-20231031132826/state -type f -exec sh -c 'rm -f "${1#/backup-20231031132826/state}"' sh {} \; &&manticore-backup --force --backup-dir=/ --restore=backup-20231031132826

Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore Backup version: 1.1.7
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore config file:
Backup dir: /
2023-11-02 15:44:10 [Info] Stored versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-02 15:44:10 [Info] Current versions: {"manticore":"6.2.13 590d63bf8@231102 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-02 15:44:10 [Info] Warning: You are trying to restore a backup from a different version. Use --force to bypass this restriction.
2023-11-02 15:44:10 [Info] Starting to restore...

Manticore config
  endpoint =  http://172.21.0.4:9312
2023-11-02 15:44:10 [Info] Restoring config files...
2023-11-02 15:44:10 [Info]   config files - OK
2023-11-02 15:44:10 [Info] Restoring state files...
2023-11-02 15:44:10 [Info]   state files - OK
2023-11-02 15:44:10 [Info] Restoring data files...
2023-11-02 15:44:10 [Info]   tables' files - OK
2023-11-02 15:44:10 [Info] The backup '//backup-20231031132826' was successfully restored.
2023-11-02 15:44:10 [Info] Elapsed time: 0.03s
2023-11-02 15:44:11 [Info] Done

root@9981475da986:/# ls /var/lib/manticore/
manticore.json  v.meta  v.ram  v.settings

Run in our linux_dockers in ubuntu22jammy

donhardman commented 10 months ago

I found the issue and it's due to root directory, should b fixed in 1.1.9, please try again

djklim87 commented 10 months ago

Yes now all fine

Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore Backup version: 1.1.9
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)

Manticore config file: 
Backup dir: /
2023-11-03 07:42:34 [Info] Stored versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-03 07:42:34 [Info] Current versions: {"manticore":"6.2.13 bd6b489b3@231102 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-03 07:42:34 [Info] Warning: You are trying to restore a backup from a different version. Use --force to bypass this restriction.
2023-11-03 07:42:34 [Info] Starting to restore...

Manticore config
  endpoint =  http://172.21.0.10:9312
2023-11-03 07:42:34 [Info] Restoring config files...
2023-11-03 07:42:34 [Info]   config files - OK
2023-11-03 07:42:34 [Info] Restoring state files...
2023-11-03 07:42:34 [Info]   state files - OK
2023-11-03 07:42:34 [Info] Restoring data files...
2023-11-03 07:42:34 [Info]   tables' files - OK
2023-11-03 07:42:34 [Info] The backup '/backup-20231031132826' was successfully restored.
2023-11-03 07:42:34 [Info] Elapsed time: 0.03s
2023-11-03 07:42:34 [Info] Done

root@499d4c7c71b4:/# ls /var/lib/manticore/v
v.meta  v.ram  v.settings