search

tbarbugli / cassandra_snapshotter

A tool to backup cassandra nodes using snapshots and incremental backups on S3
Other
222 stars 122 forks source link

Failure to backup 30GB keyspace #128

Open rchannel opened 5 years ago

rchannel commented 5 years ago

[172.18.98.145] out: STRUCTURED: time=2018-09-21T04:21:46.718135-00 pid=27117 [172.18.98.145] out: cassandra_snapshotter.agent INFO MSG: Initialized multipart upload for file /var/lib/cassandra/data/prodna/service_providers_by_sp_name-6fa6bac0931a11e7825d3554ea785613/snapshots/20180921040019/mc-62-big-Data.db to Production-NA/2018092104 Fatal error: run() received nonzero return code -1 while executing!

Requested: cassandra-snapshotter-agent put --s3-bucket-name=prodna-cassandra-backups-us-west-2 --s3-bucket-region=us-east-1 --s3-ssenc --s3-base-path=Production_NA/20180921040019/172.18.108.32 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4 Executed: /bin/bash -l -c "cassandra-snapshotter-agent put --s3-bucket-name=prodna-cassandra-backups-us-west-2 --s3-bucket-region=us-east-1 --s3-ssenc --s3-base-path=Production_NA/20180921040019/172.18.108.32 --manifest=/tmp/backupmanifest --bufsize=64 --concurrency=4"

Aborting

This is one of several huge keyspaces that does not complete.

rchannel commented 5 years ago

Found I can backup all but the ring if I run the backup locally on the machines but still unable to use to backup all nodes without failures on large cassandra installs.