search

emoncms / backup

emoncms backup module
GNU Affero General Public License v3.0
1 stars 14 forks source link

Add hostname and error handling to the export process #73

Closed CDuffers closed 12 months ago

CDuffers commented 1 year ago

This PR was triggered to support issue https://github.com/emoncms/backup/issues/63, provide error checking and alerting and refine the log update window to support this. It also build on top of PR https://github.com/emoncms/backup/pull/70

As this is such an important part of the tool and I've compared checksums of the original backup and then after imports and exports to ensure that nothing has changed between them beyond something expected or that can be justified.

The originating export was taken from a system with Emonbase and EmonTX. My test setup is a VM running Ubuntu 22.10.

The procedure followed was to:

  1. Build a vanilla VM with the latest code
  2. Import my original export - /mnt/vfs_share/emoncms-backup-2023-02-23.tar.gz below
  3. Run an export, this failed due to insufficient space - handy for proving the error checking
  4. Re-run export (Export 1) - ~/Downloads/emoncms-backup-emoncms-vm-2023-03-26.tar.gz below
  5. Import the Export 1
  6. Run export (Export 2) - ~/Downloads/emoncms-backup-emoncms-vm-2023-03-26-1.tar.gz below

I started by untarring the original export and collating a list of the file extracted:

pi@clive-ThinkPad-T470:~$ cd /tmp
pi@clive-ThinkPad-T470:/tmp$ rm -rf new* orig*
pi@clive-ThinkPad-T470:/tmp$ mkdir orig new new2
pi@clive-ThinkPad-T470:/tmp$ tar -xvf /mnt/vfs_share/emoncms-backup-2023-02-23.tar.gz -C /tmp/orig 
emoncms.sql
emonhub.conf
settings.ini
phpfina/
phpfina/19.meta
phpfina/9.dat
phpfina/62.meta
phpfina/20.dat
phpfina/50.meta
phpfina/22.meta
phpfina/48.dat
phpfina/12.meta
phpfina/10.dat
phpfina/11.meta
phpfina/17.meta
phpfina/8.dat
phpfina/62.dat
phpfina/20.meta
phpfina/7.dat
phpfina/22.dat
phpfina/14.dat
phpfina/17.dat
phpfina/18.dat
phpfina/14.meta
phpfina/11.dat
phpfina/12.dat
phpfina/50.dat
phpfina/13.dat
phpfina/48.meta
phpfina/19.dat
phpfina/21.dat
phpfina/30.meta
phpfina/13.meta
phpfina/7.meta
phpfina/21.meta
phpfina/30.dat
phpfina/10.meta
phpfina/18.meta
phpfina/9.meta
phpfina/8.meta
phpfiwa/
phptimeseries/
phptimeseries/feed_61.MYD
phptimeseries/feed_26.MYD
phptimeseries/feed_27.MYD
phptimeseries/feed_24.MYD
phptimeseries/feed_29.MYD
phptimeseries/feed_28.MYD
phptimeseries/feed_25.MYD
phptimeseries/feed_31.MYD
pi@clive-ThinkPad-T470:/tmp$ find orig/ | sed "s;orig/;;" | grep -v "^$" | sort > /tmp/orig.files
pi@clive-ThinkPad-T470:/tmp$

Then untar export 1 and collate a list of files:

pi@clive-ThinkPad-T470:/tmp$ tar -xvf ~/Downloads/emoncms-backup-emoncms-vm-2023-03-26.tar.gz -C /tmp/new
emoncms.sql
settings.ini
phpfina/
phpfina/14.dat
phpfina/11.dat
phpfina/10.dat
phpfina/21.dat
phpfina/62.meta
phpfina/14.meta
phpfina/18.dat
phpfina/8.meta
phpfina/18.meta
phpfina/48.dat
phpfina/19.dat
phpfina/17.meta
phpfina/17.dat
phpfina/9.dat
phpfina/22.meta
phpfina/13.meta
phpfina/7.meta
phpfina/22.dat
phpfina/7.dat
phpfina/9.meta
phpfina/30.meta
phpfina/13.dat
phpfina/10.meta
phpfina/21.meta
phpfina/12.dat
phpfina/19.meta
phpfina/8.dat
phpfina/20.meta
phpfina/11.meta
phpfina/62.dat
phpfina/30.dat
phpfina/20.dat
phpfina/50.meta
phpfina/48.meta
phpfina/50.dat
phpfina/12.meta
phpfiwa/
phptimeseries/
phptimeseries/feed_26.MYD
phptimeseries/feed_27.MYD
phptimeseries/feed_61.MYD
phptimeseries/feed_25.MYD
phptimeseries/feed_24.MYD
phptimeseries/feed_28.MYD
phptimeseries/feed_31.MYD
phptimeseries/feed_29.MYD
pi@clive-ThinkPad-T470:/tmp$ find new/ | sed "s;new/;;" | grep -v "^$" | sort > /tmp/new.files

Run a diff between the 2 to compare if files are missing

pi@clive-ThinkPad-T470:/tmp$ diff /tmp/orig.files /tmp/new.files
2d1
< emonhub.conf
pi@clive-ThinkPad-T470:/tmp$

emonhub.conf not found, this is expected as my test VM doesn't have emonhub installed

Create a list of MD5 checksums for all files in the original and export 1 directories, then compare the checksums:

pi@clive-ThinkPad-T470:/tmp$ $(cd orig && find . -type f | xargs md5sum > /tmp/orig.sum)
pi@clive-ThinkPad-T470:/tmp$ $(cd new && find . -type f | xargs md5sum > /tmp/new.sum)
pi@clive-ThinkPad-T470:/tmp$ diff orig.sum new.sum
10c10
< cecf3bc7b55f66fd7f8625af5f4bd19d  ./emoncms.sql
---
> e3d340ca7b6b9e72bd1e6b464e29f1c3  ./emoncms.sql
47d46
< 9e96449545263c5e76e3323665521d9b  ./emonhub.conf
pi@clive-ThinkPad-T470:/tmp$

Emonhub expected as it's not installed on my VM and thereby not backed up.

Run a diff on the 2 emoncms.sql:

pi@clive-ThinkPad-T470:/tmp$ diff orig/emoncms.sql new/emoncms.sql
1c1
< -- MySQL dump 10.17  Distrib 10.3.17-MariaDB, for debian-linux-gnueabihf (armv7l)
---
> -- MariaDB dump 10.19  Distrib 10.6.12-MariaDB, for debian-linux-gnu (x86_64)
5c5
< -- Server version 10.3.17-MariaDB-0+deb10u1
---
> -- Server version 10.6.12-MariaDB-0ubuntu0.22.10.1
28c28
< ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
65c65
< ) ENGINE=MyISAM AUTO_INCREMENT=12 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=12 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
88c88
< ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
---
> ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci;
117c117
< ) ENGINE=MyISAM AUTO_INCREMENT=21 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=21 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
152c152
< ) ENGINE=MyISAM AUTO_INCREMENT=63 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=63 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
178c178
< ) ENGINE=MyISAM AUTO_INCREMENT=22 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=22 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
208c208
< ) ENGINE=MyISAM AUTO_INCREMENT=529 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=529 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
234c234
< ) ENGINE=MyISAM AUTO_INCREMENT=14 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=14 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
257c257
< ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
282c282
< ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
310c310
< ) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
332c332
< ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
357c357
< ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
399c399
< ) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
---
> ) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci;
421c421
< -- Dump completed on 2023-02-23 20:40:30
---
> -- Dump completed on 2023-03-26 22:02:28
pi@clive-ThinkPad-T470:/tmp$

The changes are due to updated software and add COLLATE or DEFAULT CHARSET to lines and different dump completion times, hence nothing untoward.

Now untar export 2 and compare file contents with export 1:

pi@clive-ThinkPad-T470:/tmp$ tar -xvf ~/Downloads/emoncms-backup-emoncms-vm-2023-03-26-1.tar.gz -C /tmp/new2
emoncms.sql
settings.ini
phpfina/
phpfina/14.dat
phpfina/11.dat
phpfina/10.dat
phpfina/21.dat
phpfina/62.meta
phpfina/14.meta
phpfina/18.dat
phpfina/8.meta
phpfina/18.meta
phpfina/48.dat
phpfina/19.dat
phpfina/17.meta
phpfina/17.dat
phpfina/9.dat
phpfina/22.meta
phpfina/13.meta
phpfina/7.meta
phpfina/22.dat
phpfina/7.dat
phpfina/9.meta
phpfina/30.meta
phpfina/13.dat
phpfina/10.meta
phpfina/21.meta
phpfina/12.dat
phpfina/19.meta
phpfina/8.dat
phpfina/20.meta
phpfina/11.meta
phpfina/62.dat
phpfina/30.dat
phpfina/20.dat
phpfina/50.meta
phpfina/48.meta
phpfina/50.dat
phpfina/12.meta
phpfiwa/
phptimeseries/
phptimeseries/feed_26.MYD
phptimeseries/feed_27.MYD
phptimeseries/feed_61.MYD
phptimeseries/feed_25.MYD
phptimeseries/feed_24.MYD
phptimeseries/feed_28.MYD
phptimeseries/feed_31.MYD
phptimeseries/feed_29.MYD
pi@clive-ThinkPad-T470:/tmp$ find new2/ | sed "s;new2/;;" | grep -v "^$" | sort > /tmp/new2.files
pi@clive-ThinkPad-T470:/tmp$ diff /tmp/new.files /tmp/new2.files
pi@clive-ThinkPad-T470:/tmp$

No changes here.

Create checksums for export 2 and compare with export 1:

pi@clive-ThinkPad-T470:/tmp$ $(cd new2 && find . -type f | xargs md5sum > /tmp/new2.sum)
pi@clive-ThinkPad-T470:/tmp$ diff new.sum new2.sum
10c10
< e3d340ca7b6b9e72bd1e6b464e29f1c3  ./emoncms.sql
---
> 254087cddb158a792b5bb94e5a64fb11  ./emoncms.sql
pi@clive-ThinkPad-T470:/tmp$

Only emoncms.sql different as follows:

pi@clive-ThinkPad-T470:/tmp$ diff new/emoncms.sql new2/emoncms.sql
421c421
< -- Dump completed on 2023-03-26 22:02:28
---
> -- Dump completed on 2023-03-26 22:21:51
pi@clive-ThinkPad-T470:/tmp$

The only delta being the time it ran.

Output from failed export in step 3:

=== Emoncms export start ===
Sun 26 Mar 22:01:23 BST 2023
Backup module version:
    "version"      : "2.3.2",
EUID: 1000
Reading /opt/emoncms/modules/backup/config.cfg....
Location of databases: /var/opt/emoncms
Location of emonhub.conf: /etc/emonhub
Location of Emoncms: /var/www/emoncms
Backup destination: /var/opt/emoncms/backup
emoncms backup module location /var/www/emoncms/Modules/backup
-- adding /var/opt/emoncms/backup/emoncms.sql to archive --
tar: Removing leading `/' from member names
/var/opt/emoncms/backup/emoncms.sql
tar: Removing leading `/' from hard link targets
no /etc/emonhub/emonhub.conf to backup
-- adding /var/www/emoncms/settings.ini to archive --
tar: Removing leading `/' from member names
/var/www/emoncms/settings.ini
tar: Removing leading `/' from hard link targets
no /var/www/emoncms/settings.php to backup
-- adding /var/opt/emoncms/phpfina to archive --
phpfina/
phpfina/14.dat
phpfina/11.dat
phpfina/10.dat
phpfina/21.dat
phpfina/62.meta
phpfina/14.meta
phpfina/18.dat
phpfina/8.meta
phpfina/18.meta
phpfina/48.dat
tar: /var/opt/emoncms/backup/emoncms-backup-emoncms-vm-2023-03-26.tar: Wrote only 8192 of 10240 bytes
tar: Error is not recoverable: exiting now
Error: RC=2 occurred on line 143
Error: failed to tar phpfina
-- adding /var/opt/emoncms/phpfiwa to archive --
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
Error: RC=2 occurred on line 143
Error: failed to tar phpfiwa
-- adding /var/opt/emoncms/phptimeseries to archive --
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
Error: RC=2 occurred on line 143
Error: failed to tar phptimeseries
Compressing archive...
/var/opt/emoncms/backup/emoncms-backup-emoncms-vm-2023-03-26.tar:   
gzip: /var/opt/emoncms/backup/emoncms-backup-emoncms-vm-2023-03-26.tar.gz: No space left on device
Error: RC=1 occurred on line 156
Error: failed to compress tar file
emoncms export failed
=== Emoncms export completed with ERRORS! ===

And output from successful run in step 4:


=== Emoncms export start ===
Sun 26 Mar 22:02:28 BST 2023
Backup module version:
    "version"      : "2.3.2",
EUID: 1000
Reading /opt/emoncms/modules/backup/config.cfg....
Location of databases: /var/opt/emoncms
Location of emonhub.conf: /etc/emonhub
Location of Emoncms: /var/www/emoncms
Backup destination: /var/opt/emoncms/backup
emoncms backup module location /var/www/emoncms/Modules/backup
-- adding /var/opt/emoncms/backup/emoncms.sql to archive --
tar: Removing leading `/' from member names
/var/opt/emoncms/backup/emoncms.sql
tar: Removing leading `/' from hard link targets
no /etc/emonhub/emonhub.conf to backup
-- adding /var/www/emoncms/settings.ini to archive --
tar: Removing leading `/' from member names
/var/www/emoncms/settings.ini
tar: Removing leading `/' from hard link targets
no /var/www/emoncms/settings.php to backup
-- adding /var/opt/emoncms/phpfina to archive --
phpfina/
phpfina/14.dat
phpfina/11.dat
phpfina/10.dat
phpfina/21.dat
phpfina/62.meta
phpfina/14.meta
phpfina/18.dat
phpfina/8.meta
phpfina/18.meta
phpfina/48.dat
phpfina/19.dat
phpfina/17.meta
phpfina/17.dat
phpfina/9.dat
phpfina/22.meta
phpfina/13.meta
phpfina/7.meta
phpfina/22.dat
phpfina/7.dat
phpfina/9.meta
phpfina/30.meta
phpfina/13.dat
phpfina/10.meta
phpfina/21.meta
phpfina/12.dat
phpfina/19.meta
phpfina/8.dat
phpfina/20.meta
phpfina/11.meta
phpfina/62.dat
phpfina/30.dat
phpfina/20.dat
phpfina/50.meta
phpfina/48.meta
phpfina/50.dat
phpfina/12.meta
-- adding /var/opt/emoncms/phpfiwa to archive --
phpfiwa/
-- adding /var/opt/emoncms/phptimeseries to archive --
phptimeseries/
phptimeseries/feed_26.MYD
phptimeseries/feed_27.MYD
phptimeseries/feed_61.MYD
phptimeseries/feed_25.MYD
phptimeseries/feed_24.MYD
phptimeseries/feed_28.MYD
phptimeseries/feed_31.MYD
phptimeseries/feed_29.MYD
Compressing archive...
/var/opt/emoncms/backup/emoncms-backup-emoncms-vm-2023-03-26.tar:    78.3% -- replaced with /var/opt/emoncms/backup/emoncms-backup-emoncms-vm-2023-03-26.tar.gz
Backup saved: /var/opt/emoncms/backup/emoncms-backup-emoncms-vm-2023-03-26.tar.gz
Sun 26 Mar 22:03:39 BST 2023

Backup included components: /var/opt/emoncms/backup/emoncms.sql /var/www/emoncms/settings.ini phpfina phpfiwa phptimeseries
INFO: These components couldn't be found to backup: /etc/emonhub/emonhub.conf /var/www/emoncms/settings.php

Export finished...refresh page to view download link

=== Emoncms export complete! ===
TrystanLea commented 12 months ago

Thanks @CDuffers apologies for the delay!