Closed djklim87 closed 10 months ago
versions.json in this case:
{"manticore":"6.0.5 f77ce0e65@230524 dev","columnar":"0.0.0","secondary":"0.0.0"}
Decided that we won't allow to restore in a version different from the one the backup was made in. There should be flag --force
or such to force restore.
Decided on call that we're going to follow this logic:
searchd -v
--force
ignores all the aboveImplemented in this pull: https://github.com/manticoresoftware/manticoresearch-backup/pull/85
Implemented in this pull: #85
And how I can test it? Maybe should you merge this into master and build deb/rpm package?
I believe it's been waiting for my review.
https://github.com/manticoresoftware/manticoresearch-backup/pull/85 didn't pass the tests. Reassigning to @donhardman to look into that.
No it's still not working
2023-10-31T12:26:06.151872211Z + manticore-backup --version
2023-10-31T12:26:06.189049352Z Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
2023-10-31T12:26:06.189094711Z
2023-10-31T12:26:06.200092228Z Manticore Backup version: 1.1.1
2023-10-31T12:26:06.206261426Z + manticore-backup --force --backup-dir=/tmp/ --restore=backup
2023-10-31T12:26:06.237071852Z Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
2023-10-31T12:26:06.237125927Z
2023-10-31T12:26:06.247807871Z Manticore config file:
2023-10-31T12:26:06.247846228Z Backup dir: /tmp/
2023-10-31T12:26:06.262314505Z 2023-10-31 12:26:06 [Info] Stored versions: {"manticore":"6.0.5 f77ce0e65@230524 dev","columnar":"0.0.0","secondary":"0.0.0"}
2023-10-31T12:26:06.262356709Z 2023-10-31 12:26:06 [Info] Current versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-10-31T12:26:06.262386221Z 2023-10-31 12:26:06 [Info] Warning: You try to restore backup of the different version. Use --force to bypass this.
2023-10-31T12:26:06.262396419Z 2023-10-31 12:26:06 [Info] Starting to restore...
2023-10-31T12:26:06.286697452Z
2023-10-31T12:26:06.286742380Z Manticore config
2023-10-31T12:26:06.286753981Z endpoint = http://127.0.0.1:9308
2023-10-31T12:26:06.489484274Z 2023-10-31 12:26:06 [Info] Restoring config files...
2023-10-31T12:26:06.490296417Z 2023-10-31 12:26:06 [Info] config files - OK
2023-10-31T12:26:06.490340594Z 2023-10-31 12:26:06 [Info] Restoring state files...
2023-10-31T12:26:06.603427020Z 2023-10-31 12:26:06 [Error] Failed to create directory – "/usr/share/manticore/modules/manticore-buddy/vendor/composer/composer/doc/fixtures/repo-composer-with-providers/p/bar"
As per log I can see that it works fine. The issue is one of two:
I need more information to check it out. Like permissions on directories, exactly steps to reproduce and so on.
Yes maybe you right. I tried to remove this files, start again and got new error:
2023-10-31T14:19:52.460561446Z + ls /usr/share/manticore/modules/manticore-buddy/vendor/
2023-10-31T14:19:52.463271950Z bin evenement justinrainbow php-ds react seld
2023-10-31T14:19:52.463308998Z composer fig manticoresoftware psr ringcentral symfony
2023-10-31T14:19:52.463634979Z + manticore-backup --version
2023-10-31T14:19:52.504617802Z Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
2023-10-31T14:19:52.504674005Z
2023-10-31T14:19:52.515813027Z Manticore Backup version: 1.1.1
2023-10-31T14:19:52.524603331Z + manticore-backup --force --backup-dir=/ --restore=docker-entrypoint-initdb.d
2023-10-31T14:19:52.565568030Z Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
2023-10-31T14:19:52.565641696Z
2023-10-31T14:19:52.577415487Z Manticore config file:
2023-10-31T14:19:52.577460920Z Backup dir: /
2023-10-31T14:19:52.598381462Z 2023-10-31 14:19:52 [Info] Stored versions: {"manticore":"6.0.5 f77ce0e65@230524 dev","columnar":"0.0.0","secondary":"0.0.0"}
2023-10-31T14:19:52.598465056Z 2023-10-31 14:19:52 [Info] Current versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-10-31T14:19:52.598492827Z 2023-10-31 14:19:52 [Info] Warning: You try to restore backup of the different version. Use --force to bypass this.
2023-10-31T14:19:52.598512193Z 2023-10-31 14:19:52 [Info] Starting to restore...
2023-10-31T14:19:52.628040172Z
2023-10-31T14:19:52.628094583Z Manticore config
2023-10-31T14:19:52.628106164Z endpoint = http://127.0.0.1:9308
2023-10-31T14:19:52.942103941Z 2023-10-31 14:19:52 [Info] Restoring config files...
2023-10-31T14:19:52.943324840Z 2023-10-31 14:19:52 [Info] config files - OK
2023-10-31T14:19:52.943360616Z 2023-10-31 14:19:52 [Info] Restoring state files...
2023-10-31T14:19:53.047612690Z 2023-10-31 14:19:53 [Error] Failed to create directory – "usr/share/manticore/modules/manticore-buddy/vendor/psr/log"
it looks like now it is trying to create a new path not from the system root folder
@djklim87 how do I reproduce this from scratch? I can't reproduce it like this:
Install release:
root@b3e00313b77e:/# wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore manticore-extra
Searchd version:
root@b3e00313b77e:/# searchd -v
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
...
Create a test table and backup:
root@b3e00313b77e:/# systemctl start manticore
root@b3e00313b77e:/# mysql -P9306 -h0 -e "create table t"
root@b3e00313b77e:/# manticore-backup --backup-dir=/tmp/
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
Manticore config file: /etc/manticoresearch/manticore.conf
Tables to backup: all tables
Backup dir: /tmp/
Manticore config
endpoint = http://127.0.0.1:9312
Manticore versions:
manticore: 6.2.12 dc5144d35@230822
columnar: 2.2.4 5aec342@230822
secondary: 2.2.4 5aec342@230822
Manticore versions:
manticore: 6.2.12 dc5144d35@230822
columnar: 2.2.4 5aec342@230822
secondary: 2.2.4 5aec342@230822
2023-10-31 16:03:11 [Info] Starting the backup...
2023-10-31 16:03:11 [Info] Backing up config files...
2023-10-31 16:03:11 [Info] config files - OK
2023-10-31 16:03:11 [Info] Backing up global state files...
2023-10-31 16:03:11 [Info] global state files – OK
2023-10-31 16:03:11 [Info] Backing up tables...
2023-10-31 16:03:11 [Info] t (rt) [481B]...
2023-10-31 16:03:11 [Info] OK
2023-10-31 16:03:11 [Info] Running sync
2023-10-31 16:03:11 [Info] OK
2023-10-31 16:03:11 [Info] You can find backup here: /tmp/backup-20231031160311
2023-10-31 16:03:11 [Info] Elapsed time: 0.32s
2023-10-31 16:03:12 [Info] Done
2023-10-31 16:03:12 [Info]
Unfreezing all tables...
2023-10-31 16:03:12 [Info] t...
2023-10-31 16:03:12 [Info] OK
Upgrade Manticore to a dev version:
wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && \
sudo dpkg -i manticore-dev-repo.noarch.deb && \
sudo apt -y update && \
sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l
searchd version:
root@b3e00313b77e:/# searchd -v
Manticore 6.2.13 3e7105823@231031 dev (columnar 2.2.5 b8be4eb@230928) (secondary 2.2.5 b8be4eb@230928)
Restore:
root@b3e00313b77e:/# systemctl stop manticore
root@b3e00313b77e:/# manticore-backup --backup-dir=/tmp/ --restore
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
Manticore config file:
Backup dir: /tmp/
Available backups: 1
backup-20231031160311 (Oct 31 2023 16:03:11)
root@b3e00313b77e:/# rm /etc/manticoresearch/manticore.conf /var/lib/manticore/manticore.json
root@b3e00313b77e:/# manticore-backup --backup-dir=/tmp/ --restore=backup-20231031160311 --force
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
Manticore config file:
Backup dir: /tmp/
2023-10-31 16:07:35 [Info] Stored versions: {"manticore":"6.2.12 dc5144d35@230822 ","columnar":"2.2.4 5aec342@230822","secondary":"2.2.4 5aec342@230822"}
2023-10-31 16:07:35 [Info] Current versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-10-31 16:07:35 [Info] Warning: You are trying to restore a backup from a different version. Use --force to bypass this restriction.
2023-10-31 16:07:35 [Info] Starting to restore...
Manticore config
endpoint = http://127.0.0.1:9312
2023-10-31 16:07:35 [Info] Restoring config files...
2023-10-31 16:07:35 [Info] config files - OK
2023-10-31 16:07:35 [Info] Restoring state files...
2023-10-31 16:07:35 [Info] state files - OK
2023-10-31 16:07:35 [Info] Restoring data files...
2023-10-31 16:07:35 [Info] tables' files - OK
2023-10-31 16:07:35 [Info] The backup '/tmp/backup-20231031160311' was successfully restored.
2023-10-31 16:07:35 [Info] Elapsed time: 0.03s
2023-10-31 16:07:35 [Info] Done
root@b3e00313b77e:/#
No, problem still available I tested at our linux_dockers
apt-get update && apt-get install -y wget gnupg unzip && \
wget https://repo.manticoresearch.com/manticore-dev-repo.noarch.deb && sudo dpkg -i manticore-dev-repo.noarch.deb && sudo apt -y update && sudo apt -y install manticore manticore-extra manticore-common manticore-server manticore-server-core manticore-tools manticore-executor manticore-buddy manticore-backup manticore-columnar-lib manticore-server-core-dbgsym manticore-tools-dbgsym manticore-columnar-lib-dbgsym manticore-icudata-65l && \
wget https://github.com/manticoresoftware/manticoresearch-backup/files/12859973/backup-20231009160538.zip && unzip backup-20231009160538.zip && \
manticore-backup --version && \
find /backup-20231009160538/config -type f -exec sh -c 'rm -f "${1#/backup-20231009160538/config}"' sh {} \; &&\
find /backup-20231009160538/state -type f -exec sh -c 'rm -f "${1#/backup-20231009160538/state}"' sh {} \; &&\
manticore-backup --force --backup-dir=/ --restore=backup-20231009160538
Result
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
Manticore Backup version: 1.1.1
Copyright (c) 2023, Manticore Software LTD (https://manticoresearch.com)
Manticore config file:
Backup dir: /
2023-11-01 12:03:37 [Info] Stored versions: {"manticore":"6.0.5 f77ce0e65@230524 dev","columnar":"0.0.0","secondary":"0.0.0"}
2023-11-01 12:03:37 [Info] Current versions: {"manticore":"6.2.13 3e7105823@231031 dev","columnar":"2.2.5 b8be4eb@230928","secondary":"2.2.5 b8be4eb@230928"}
2023-11-01 12:03:37 [Info] Warning: You are trying to restore a backup from a different version. Use --force to bypass this restriction.
2023-11-01 12:03:37 [Info] Starting to restore...
Manticore config
endpoint = http://127.0.0.1:9308
2023-11-01 12:03:37 [Info] Restoring config files...
2023-11-01 12:03:37 [Info] config files - OK
2023-11-01 12:03:37 [Info] Restoring state files...
2023-11-01 12:03:37 [Error] Failed to create directory – "usr/share/manticore"
The problem arises from an old backup that was created with an incompatible version. We plan to rectify this in future releases by also storing the backup version. However, for now, we recommend creating and restoring backups using the same Manticore-backup version.
Looks like all fine now
I faced with interesting behavior.
If I created a dump with the old version of Buddy, I can't restore it because I haven't desired path.
For example in my dump has files inside
/usr/share/manticore/modules/manticore-buddy/vendor/composer/composer/doc/...
folder In newer versions, this folder doesn't exist. When I'll try to restore this backup I get an errorI guess it's because we don't create folders, we just move files from the archive recursively.