search

JeremyGrosser / tablesnap

Uses inotify to monitor Cassandra SSTables and upload them to S3
BSD 2-Clause "Simplified" License
181 stars 86 forks source link

Boto3 for just tablesnap, tokens and KMS encryption #89

Open juiceblender opened 7 years ago

juiceblender commented 7 years ago

Hi guys,

Just thought I would put up another PR this time. This PR was born out of my need to able to use KMS to encrypt backups instead of just SSE but it doesn't look like I am able to do that using boto. So I bumped it to boto3 for just tablesnap only. Boto and boto3 can coexist during the migration so I figured this would be a start: http://boto3.readthedocs.io/en/latest/guide/migration.html

Functionally I've tested md5-on-start (just to be sure etags are right with multiparts...it may not have worked before..?), KMS, SSE, and generating tokens and of course, taking the backup. Let me know if there are any questions/issues/suggestions!

Thanks.

JeremyGrosser commented 7 years ago

I've been wanting to do this for a while, but I'm a bit afraid to introduce a dependency on boto3 as there is no system package for it in Debian jessie and quite a few people (including our unit test scripts) haven't moved to stretch yet.

Generally speaking, I think we should probably start working towards a backwards-incompatible major release that moves completely to Python 3 with boto3 and drops support for older systems completely... Ideally that release would also be tested against Cassandra 3.

JeremyGrosser commented 7 years ago

...looking into it a bit more, there is a python-botocore package in jessie that contains most of the code from boto3. Maybe we can use that? https://github.com/boto/botocore

juiceblender commented 7 years ago

Hi Jeremy,

Sorry you lost me there, how do you mean there is no system package for boto3 in Debian Jessie? If you do an install of boto3 on Debian Jessie it wouldn't work?

I have actually been running my tests on Cassandra 3...