search

PowerDNS / lightningstream

Lightning Stream syncs LMDB databases through S3 buckets between multiple servers, including PowerDNS Authoritative server 4.8+ LMDBs
https://doc.powerdns.com/lightningstream/
MIT License
28 stars 16 forks source link

docs: detail on integrity checks #65

Open udf2457 opened 1 year ago

udf2457 commented 1 year ago

Looking through the codebase, it looks like lightningstream does not use the Content-MD5 header in S3-compatible APIs in order to ensure end-to-end integrity of blobs uploaded.

It would be nice to have a couple of paragraphs in the docs as to how lightningstream approaches this.

wojas commented 1 year ago

Lightning Stream uses https://github.com/PowerDNS/simpleblob for object storage, which uses https://github.com/minio/minio-go for S3 storage.

udf2457 commented 1 year ago

Sure, I looked briefly through that code yesterday. However you need to explicitly set the MD5 option in the minio-go struct when calling it and - as far as I can tell after a quick glance - simpleblob is neither calculating the MD5 or calling minio-go with the MD5 option.

Sending Content-MD5 is highly recommended, because otherwise the receiving S3 service has no way of ensuring integrity. With Content-MD5, the S3 service caluclates MD5 and compares it against the MD5 provided in Content-MD5, and so you therefore have a high level of confidence that what you uploaded is what was stored.

wojas commented 1 year ago

Thanks, I added PowerDNS/simpleblob#49 for this and added it to the next patch release milestone.

udf2457 commented 1 year ago

Thanks @wojas !