bioneos / alcove

Our Unix backup system redesigned with Node.js
ISC License
0 stars 6 forks source link

Update data models #127

Open sgdavis1 opened 2 years ago

sgdavis1 commented 2 years ago

In the research and planning for #111 and #93, we have identified that there is an unnecessary redundancy between the database data models and the memory models used to manage the backup system in lib/system.js:machines:17 and lib/system.js:prepareMachines():145-164. The loose typing of JS has caused the problems mentioned above because of inconsistencies in the way the system.js reads and creates this models at init() and the way they are created when a backup is completed during active processing by the rsync.js module. They are also lightly documented and hard to determine expectations for the codebase because this documentation is not centralized anywhere.

As a solution we will rethink our database and move all of these memory models into the database. This process will require a codebase reorganization but will result in a much cleaner and more maintainable system. Use this issue to describe the plan for this work.

sgdavis1 commented 2 years ago

Current structure:

backupEvent: {
    'machine': DataTypes.STRING,
    'schedule' : DataTypes.STRING,
    'bucket' : DataTypes.STRING,  // Not sure what this is
    'rsyncExitCode' : DataTypes.INTEGER,
    'rsyncExitReason' : DataTypes.STRING,
    'transferSize' : DataTypes.INTEGER,
    'transferTimeSec' : DataTypes.INTEGER,
    'dir' : DataTypes.STRING,
    'backupTime': (timestamp)
}

sizes: {
    'machine' : DataTypes.STRING,
    'location' : DataTypes.STRING,
    'size' : DataTypes.INTEGER
}

No current foreign key relationships exist. We will at least need a machine object added to this and both of these will be adjusted. Suggest the new schema below.

Jacks-01 commented 2 years ago

A couple questions,

No current foreign key relationships exist. We will at least need a machine object added to this and both of these will be adjusted. Suggest the new schema below.

so the machine property exists in sizes.js, backupEvents.js, and system.js. When you say you want a new machine object, are you implying we should create a new machine model and scrap the old ones? It's somewhat confusing because machine is referred to in many different places.

On a side note, would creating an ER diagram be helpful for documentation / clarification?

ashleyseo commented 2 years ago

Here is what we were thinking about so far, do you have any suggestions?

`backupEvent: { 'machine': DataTypes.STRING, 'schedule' : DataTypes.STRING, 'rsyncExitCode' : DataTypes.INTEGER, 'rsyncExitReason' : DataTypes.STRING, 'transferSize' : DataTypes.INTEGER, 'transferTimeSec' : DataTypes.INTEGER, 'dir' : DataTypes.STRING, 'backupTime': DataTypes.STRING }

sizes: { 'machine' : DataTypes.STRING, 'location' : DataTypes.STRING, 'size' : DataTypes.INTEGER }

machine : { 'schedule' : DataTypes.STRING, 'name': DataTypes.STRING, 'host': DataTypes.STRING, 'port': DataTypes.INTEGER, 'backupDirectories': DataTypes.ARRAY, 'lastBackup': DataTypes.STRING, 'totalSize': DataTypes.INTEGER }`