gdha / rear-automated-testing

Relax-and-Recover Automated Testing
https://gdha.github.io/rear-automated-testing/
GNU General Public License v3.0
10 stars 2 forks source link

rear mkbackup failure return code (rc) not captured correctly #43

Closed gdha closed 6 years ago

gdha commented 6 years ago

On the screen we see:

Run rear -v mkbackup
Relax-and-Recover 2.2-git.0.3a9bb65.unknown / 2017-12-04
Using log file: /var/log/rear/rear-client.log

The rear mkbackup was successful

And, in the logfile we see:

2017-12-05 12:52:17.755289604 Run 'rear -v mkbackup'
Warning: Permanently added '192.168.33.10' (ECDSA) to the list of known hosts.^M
Relax-and-Recover 2.2-git.0.3a9bb65.unknown / 2017-12-04
Using log file: /var/log/rear/rear-client.log
ERROR: Bareos client status unknown on director.
Aborting due to an error, check /var/log/rear/rear-client.log for details
2017-12-05 12:52:33.558302314

The code is as follow:

Log "Run 'rear -v mkbackup'"
echo "$(bold Run 'rear -v mkbackup')"
ssh -i ../insecure_keys/vagrant.private root@$client "rear -v mkbackup" | tee -a $LOGFILE
rc=$?

LogPrint "
"
if [[ $rc -ne 0 ]] ; then

Clearly, the rc seems to be always be 0 (most probably due to the use of the | command)

gdha commented 6 years ago

The problem on the server VM was:

# cat /var/lib/bareos/bareos-dir.bareos-dir.15374952.mail
05-Dec 12:51 bareos-dir JobId 0: Fatal error: Query failed: SELECT VersionId FROM Version: ERR=ERROR:  permission denied for relation version

05-Dec 12:51 bareos-dir JobId 0: Fatal error: Could not open Catalog "MyCatalog", database "bareos".
05-Dec 12:51 bareos-dir JobId 0: Fatal error: Query failed: SELECT VersionId FROM Version: ERR=ERROR:  permission denied for relation version

05-Dec 12:51 bareos-dir ERROR TERMINATION
Please correct the configuration in /etc/bareos/bareos-dir.conf

To fix it run the following once more (should have been done by ansible/server/roles/rear-test/tasks/setup-bareos-server.yml):

su postgres -c /usr/lib/bareos/scripts/update_bareos_tables 
systemctl start bareos-dir
gdha commented 6 years ago

Modified our test script and now we capture the error:

Run rear -v mkbackup
Relax-and-Recover 2.2-git.0.3a9bb65.unknown / 2017-12-04
Using log file: /var/log/rear/rear-client.log

Please check the rear logging /var/log/rear/rear-client.log
The last 20 lines are:
/var/lib/nfs/rpc_pipefs 555 root root
/var/lib/nfs/rpc_pipefs 555 root root
2017-12-05 13:34:44.211593044 Including prep/BAREOS/default/450_check_BAREOS_client_configured.sh
2017-12-05 13:34:44.226628858 Including prep/BAREOS/default/500_check_BAREOS_bconsole_results.sh
2017-12-05 13:34:44.232399779 Skipping ping test
2017-12-05 13:34:59.248867010 ERROR: Bareos client status unknown on director.
==== Stack trace ====
Trace 0: /usr/sbin/rear:543 main
Trace 1: /usr/share/rear/lib/mkbackup-workflow.sh:9 WORKFLOW_mkbackup
Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source
Trace 4: /usr/share/rear/prep/BAREOS/default/500_check_BAREOS_bconsole_results.sh:46 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:251 StopIfError
Message: Bareos client status unknown on director.
gdha commented 6 years ago

Bumper:

$ sudo grep ERROR /export/rear-tests/logs/2017-12-05_13-39-03/rear-client-mkbackup.log 
modinfo: ERROR: Module nls_cp437 not found.
modinfo: ERROR: Module af_packet not found.
modinfo: ERROR: Module unix not found.
modinfo: ERROR: Module usbcore not found.
modinfo: ERROR: Module usbhid not found.
modinfo: ERROR: Module ide_cd not found.
modinfo: ERROR: Module uhci_hcd not found.
modinfo: ERROR: Module ehci_hcd not found.
modinfo: ERROR: Module xhci_hcd not found.
modinfo: ERROR: Module ohci_hcd not found.
modinfo: ERROR: Module zlib-inflate not found.
modinfo: ERROR: Module zlib-deflate not found.

Need to remove these, or use the last 20 lines?

gdha commented 6 years ago

We do a tail -20 before we grep for ERROR which seems to be sufficient.