search

aerospike / aerospike-tools-backup

Aerospike backup and restore utility
http://www.aerospike.com/docs/tools/backup/
Apache License 2.0
25 stars 20 forks source link

Add S3 checksum option for enhanced data integrity #88

Open peterrojs opened 7 months ago

peterrojs commented 7 months ago

Our client wants to ensure the data integrity of objects being uploaded to S3 buckets is not compromised during the upload. AWS S3 SDK has the ability to automatically create a checksum of the object on client-side to ensure data integrity during transfer. The checksum is then checked at the end of the upload.

Adding an option like --s3-checksum HASH_FUNC or similar to asbackup, would set the checksum algorithm for uploading the object with AWS SDK, hence enabling this feature. With hashes printed out to the console, we could make sure the objects arrive to the S3 bucket without compromising the integrity of the data.

Here are the links to the documentation about the S3 checksum feature:

dwelch-spike commented 6 months ago

Hi @peterrojs. We appreciate the informative request but aren't able to focus on this right now. We are actually in the middle of writing an Aerospike Golang backup library and will add this to the roadmap for that project. In the mean time a PR is welcome here or in the Go library when it goes public.