search

schnipdip / Dump

Auto back up images from a raspberry pi to a removable storage device automatically from an SD card.
GNU Affero General Public License v3.0
8 stars 1 forks source link

Alternative Backup Solution to Rsync #7

Open schnipdip opened 4 years ago

schnipdip commented 4 years ago

Basically, rebuild functionality of Rsync, however the function will allow a percentage display on the backups files. This will allow for multicore copying. Differential backup will have to be supported to prevent duplicates - checksum.

Issue with checksum - cpu intensive

Major Benefit, more information can be displayed to the user.

schnipdip commented 4 years ago

Can create a local local cache/database to store checksums to reduce checksum time

includes name of picture and checksum of the picture = photo rainbow table.

schnipdip commented 4 years ago

Do i need to perform a checksum on the files located in the destination drive? If I am creating a rainbow table then the values should exist in the cache. I would just have to do a comparison the the cache and the source checksum to add the files a dict object and then copy those new files.

schnipdip commented 4 years ago

added dump_it.py.

Contains the following classes:

class backup -> performs differential backup - spawns new process to copy each object to the destination class hashing - > hashes each image object. It spawns a new process for each object to perform a checksum class cacheTable -> creates a caching database of filename + checksum

schnipdip commented 4 years ago

Now that I am thinking about this more it doesn't make much sense to create a tailored backup solution. Instead we can continue to work with Rsync and if need be create a wrapper for Rsync to do what we need it to do. We are going to have to tailor the Raspberry Pi for performance. We can probably get rsync to multithread to increase performance and integrity checking can be done in rsync to ensure the files were transferred properly (advanced setting configuration for advanced users)