History-Research-Environment / HRE--History-Research-Environment

Main repo for HRE code
https://historyresearchenvironment.org/
GNU Affero General Public License v3.0
32 stars 6 forks source link

GUI_ProjectBackup – Backup a Project #29

Closed MichaelErichsen closed 4 years ago

MichaelErichsen commented 6 years ago

A project in TMG is a number of Firefox files. A project in HRE is a single H2 database file. This makes this much easier. An unscientific test showed that zipped H2 databases shrunk to between two thirds and a few percent of the original files. Just zipping and unzipping H2 files is the easiest solution. Another way would be to export each table to cvs before zipping, which has the advantage of being readable outside of HRE and H2.

HREferg commented 6 years ago

This is an area we had not come to a conclusion as we weren’t sure we knew all the facts.

There are 2 issues (1) what are we backing up/restoring? and (2) what are the available & best tools?

  1. The backup set is not just the H2 database. An HRE ‘fileset’ will consist of the H2 database, the ‘ancillary data’ that resides outside of H2 (in XML or JSON or whatever) and, optionally folder(s) of external data (such as image files) that the user may wish to include in the backup.
  2. I understand that the possible backup methods are (a) to use the H2 Backup command (which creates a zip file); (b) to use a native Java command (ZipOutputStream?), (c) to create a csv file, (d) others?

Personally I’d favour the zip method, as zip files are well understood, but we then also need to be able to add the other 2 parts to the zip package.

I understand the csv file means it’s readable by external methods, but suspect, since we’re using a well-populated DB, that the space savings will be much lower than using zip. Of course, one could create a csv file and then zip it, but this is a 2-stage process.

Your input would be appreciated (I note you seem to be having issues creating a csv file!

Don

From: Michael Erichsen notifications@github.com Sent: Friday, June 15, 2018 7:37 PM To: History-Research-Environment/HRE--History-Research-Environment HRE--History-Research-Environment@noreply.github.com Cc: HREferg don@ferguson.name; Assign assign@noreply.github.com Subject: [History-Research-Environment/HRE--History-Research-Environment] GUI_ProjectBackup – Backup a Project (#29)

A project in TMG is a number of Firefox files. A project in HRE is a single H2 database file. This makes this much easier. An unscientific test showed that zipped H2 databases shrunk to between two thirds and a few percent of the original files. Just zipping and unzipping H2 files is the easiest solution. Another way would be to export each table to cvs before zipping, which has the advantage of being readable outside of HRE and H2.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/History-Research-Environment/HRE--History-Research-Environment/issues/29 , or mute the thread https://github.com/notifications/unsubscribe-auth/AZq_wGgGA5Rr6iuSsh1g6XfloMVQXpI0ks5t84A0gaJpZM4UpSIC .

MichaelErichsen commented 6 years ago

It is not much extra work to do both zipped H2 plus the rest and zipped csv plus the rest, so I suggest we do both now. When we start testing with large databases we then might choose to eliminate one of them.

ricko2001 commented 6 years ago

I just wanted to remind folks about zip file issues: File and archive max size File path Unicode chars

No need for a reply....

On Sat, Jun 16, 2018, 1:19 AM Michael Erichsen notifications@github.com wrote:

It is not much extra work to do both zipped H2 plus the rest and zipped csv plus the rest, so I suggest we do both now. When we start testing with large databases we then might choose to eliminate one of them.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/History-Research-Environment/HRE--History-Research-Environment/issues/29#issuecomment-397796639, or mute the thread https://github.com/notifications/unsubscribe-auth/ATcR-PfI1bxNt0gEAqdhFkrzAmyH9B0Hks5t9L91gaJpZM4UpSIC .

HREferg commented 4 years ago

Closed - current Build of HRE performs zip of project H2 database