tbarbugli / cassandra_snapshotter

A tool to backup cassandra nodes using snapshots and incremental backups on S3
Other
222 stars 122 forks source link

Error : OSError: [Errno 13] Permission denied: while taking incremental backup : #53

Closed cassandrauser closed 8 years ago

cassandrauser commented 9 years ago

We're facing an issue when trying to take the incremental backup using cassandra_snapshotter tool.

We've a three node cluster in Amazon cloud. We've enabled incremental backup and moving it to S3. But when we try to execute the snapshotter tool it gives permission denied error :

ec2-154-421-401-82.compute-1.amazonaws.com] out: Traceback (most recent call last): [ec2-154-421-401-82.compute-1.amazonaws.com] out: File "/usr/local/bin/cassandra-snapshotter-agent", line 9, in [ec2-154-421-401-82.compute-1.amazonaws.com] out: load_entry_point('cassandra-snapshotter==0.4.0', 'console_scripts', 'cassandra-snapshotter-agent')() [ec2-154-421-401-82.compute-1.amazonaws.com] out: File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/agent.py", line 266, in main [ec2-154-421-401-82.compute-1.amazonaws.com] out: args.incremental_backups [ec2-154-421-401-82.compute-1.amazonaws.com] out: File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/agent.py", line 151, in put_from_manifest [ec2-154-421-401-82.compute-1.amazonaws.com] out: os.remove(f)

[ec2-154-421-401-82.compute-1.amazonaws.com] out: OSError: [Errno 13] Permission denied: '/cassandra_data/data/test_bkp/t-9bc68ac040d511e5bab8a38736f22d26/backups/test_bkp-t-ka-2-Statistics.db'

It look like some sort of permission error but we've give full permission to (t-9bc68ac040d511e5bab8a38736f22d26/) directory but no luck. also ran the tool from root account but still we see the same error. We also cleared all the snapshots in S3 and tried to execute the snapshotter again and it worked fine but after inserting some data and running the tool results in above error message.

tbarbugli commented 9 years ago

what is the output of ls -la /cassandra_data/data/test_bkp/t- 9bc68ac040d511e5bab8a38736f22d26/backups/test_bkp-t-ka-2-Statistics.db? Can you paste the full command that you use to run the snapshot (credential excluded of course).

On Thu, Sep 10, 2015 at 5:14 PM cassandrauser notifications@github.com wrote:

We're facing an issue when trying to take the incremental backup using cassandra_snapshotter tool.

We've a three node cluster in Amazon cloud. We've enabled incremental backup and moving it to S3. But when we try to execute the snapshotter tool it gives permission denied error :

ec2-154-421-401-82.compute-1.amazonaws.com] out: Traceback (most recent call last): [ec2-154-421-401-82.compute-1.amazonaws.com] out: File "/usr/local/bin/cassandra-snapshotter-agent", line 9, in [ec2-154-421-401-82.compute-1.amazonaws.com] out: load_entry_point('cassandra-snapshotter==0.4.0', 'console_scripts', 'cassandra-snapshotter-agent')() [ec2-154-421-401-82.compute-1.amazonaws.com] out: File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/agent.py", line 266, in main [ec2-154-421-401-82.compute-1.amazonaws.com] out: args.incremental_backups [ec2-154-421-401-82.compute-1.amazonaws.com] out: File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/agent.py", line 151, in put_from_manifest [ec2-154-421-401-82.compute-1.amazonaws.com] out: os.remove(f)

[ec2-154-421-401-82.compute-1.amazonaws.com] out: OSError: [Errno 13] Permission denied: '/cassandra_data/data/test_bkp/t-9bc68ac040d511e5bab8a38736f22d26/backups/test_bkp-t-ka-2-Statistics.db' [ec2-154-421-401-82.compute-1.amazonaws.com] out:

It look like some sort of permission error but we've give full permission to (t-9bc68ac040d511e5bab8a38736f22d26/) directory but no luck. also ran the tool from root account but still we see the same error. We also cleared all the snapshots in S3 and tried to execute the snapshotter again and it worked fine but after inserting some data and running the tool results in above error message.

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53.

cassandrauser commented 9 years ago

Full COmmand that we're using to run the snapshot

cassandra-snapshotter --aws-access-key-id=AKIAI --aws-secret-access-key=e0n+NkGK+zzB7mdya --s3-bucket-name=cassandra-backup-qa --s3-base-path=cassandra-devqa backup --hosts=ec2-154-421-401-82.compute-1.amazonaws.com,ec2-154-97-44-37.compute-1.amazonaws.com,ec2-54-22-21-181.compute-1.amazonaws.com --keyspaces test_bkp --cassandra-conf-path=/etc/cassandra --user=ubuntu --password=abc

tbarbugli commented 9 years ago

did you try using --use-sudo?

On Thu, Sep 10, 2015 at 5:23 PM cassandrauser notifications@github.com wrote:

Full COmmand that we're using to run the snapshot

cassandra-snapshotter --aws-access-key-id=AKIAI --aws-secret-access-key=e0n+NkGK+zzB7mdya --s3-bucket-name=cassandra-backup-qa --s3-base-path=cassandra-devqa backup --hosts=ec2-154-421-401-82.compute-1.amazonaws.com, ec2-154-97-44-37.compute-1.amazonaws.com, ec2-54-22-21-181.compute-1.amazonaws.com --keyspaces test_bkp --cassandra-conf-path=/etc/cassandra --user=ubuntu --password=abc

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53#issuecomment-139278870 .

cassandrauser commented 9 years ago

not exactly, but we tried to run it using root account but still we see the same permission error.

tbarbugli commented 9 years ago

I suggest you to give it a try, let me know what happens ;)

On Thu, Sep 10, 2015 at 5:27 PM cassandrauser notifications@github.com wrote:

not exactly, but we tried to run it using root account but still we see the same permission error.

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53#issuecomment-139280697 .

cassandrauser commented 9 years ago

Yes, we tried but it seems we're missing something in syntax.

cassandra-snapshotter --aws-access-key-id=AKIAI --aws-secret-access-key=e0n --s3-bucket-name=cassandra-backup-devqa --s3-base-path=cassandra-devqa backup --hosts=ec2-54-197-23-18.compute-1.amazonaws.com,ec2-54-197-24-13.compute-1.amazonaws.com,ec2-54-221-201-1812.compute-1.amazonaws.com --keyspaces=test_bkp --cassandra-conf-path=/etc/cassandra --use-sudo ubuntu

Traceback (most recent call last): File "/usr/local/bin/cassandra-snapshotter", line 9, in load_entry_point('cassandra-snapshotter==0.4.0', 'console_scripts', 'cassandra-snapshotter')() File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/main.py", line 241, in main run_backup(args) File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/main.py", line 57, in run_backup connection_pool_size=args.connection_pool_size File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/snapshotting.py", line 251, in init self.use_sudo = bool(strtobool(use_sudo)) File "/usr/lib/python2.7/distutils/util.py", line 325, in strtobool raise ValueError, "invalid truth value %r" % (val,)

ValueError: invalid truth value 'ubuntu'

tbarbugli commented 9 years ago

--use-sudo does not need a parameter, it's just a flag

On Thu, Sep 10, 2015 at 5:33 PM cassandrauser notifications@github.com wrote:

Yes, we tried but it seems we're missing something in syntax.

cassandra-snapshotter --aws-access-key-id=AKIAI --aws-secret-access-key=e0n --s3-bucket-name=cassandra-backup-devqa --s3-base-path=cassandra-devqa backup --hosts= ec2-54-197-23-18.compute-1.amazonaws.com, ec2-54-197-24-13.compute-1.amazonaws.com, ec2-54-221-201-1812.compute-1.amazonaws.com --keyspaces=test_bkp --cassandra-conf-path=/etc/cassandra --use-sudo ubuntu

Traceback (most recent call last):

File "/usr/local/bin/cassandra-snapshotter", line 9, in load_entry_point('cassandra-snapshotter==0.4.0', 'console_scripts', 'cassandra-snapshotter')() File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/main.py", line 241, in main run_backup(args) File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/main.py", line 57, in run_backup connection_pool_size=args.connection_pool_size File "/usr/local/lib/python2.7/dist-packages/cassandra_snapshotter/snapshotting.py", line 251, in init self.use_sudo = bool(strtobool(use_sudo)) File "/usr/lib/python2.7/distutils/util.py", line 325, in strtobool raise ValueError, "invalid truth value %r" % (val,)

ValueError: invalid truth value 'ubuntu'

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53#issuecomment-139282986 .

cassandrauser commented 9 years ago

We tried that without Ubuntu parameter but got an error :

cassandra-snapshotter --aws-access-key-id=AKIAI --aws-secret-access-key=e0n --s3-bucket-name=cassandra-backup-devqa --s3-base-path=cassandra-devqa backup --hosts=ec2-54-197-23-18.compute-1.amazonaws.com,ec2-54-197-24-13.compute-1.amazonaws.com,ec2-54-221-201-1812.compute-1.amazonaws.com --keyspaces=test_bkp --cassandra-conf-path=/etc/cassandra --use-sudo

Error : cassandra-snapshotter backup: error: argument --use-sudo: expected one argument

cassandrauser commented 9 years ago

Could you please tell what are we missing here. It's very urgent.

Thanks for the help

tbarbugli commented 9 years ago

--use-sudo=yes

On Thu, Sep 10, 2015 at 5:57 PM cassandrauser notifications@github.com wrote:

Could you please tell what are we missing here. It's very urgent.

Thanks for the help

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53#issuecomment-139290243 .

cassandrauser commented 9 years ago

Ops! Tired it but got the below error :( ######################### Fatal error: Needed to prompt for a connection or sudo password (host: ec2-54-97-244-137.compute-1.amazonaws.com), but input would be ambiguous in parallel mode

Aborting.

Fatal error: Needed to prompt for a connection or sudo password (host: ec2-54-21-201-182.compute-1.amazonaws.com), but input would be ambiguous in parallel mode

Aborting.

Fatal error: Needed to prompt for a connection or sudo password (host: ec2-54-17-231-18.compute-1.amazonaws.com), but input would be ambiguous in parallel mode

Aborting. ##############

tbarbugli commented 9 years ago

the command is failing because sudo is asked for a password prompt. make sure that you can use sudo on all your boxes without the password prompt

On Thu, Sep 10, 2015 at 6:07 PM cassandrauser notifications@github.com wrote:

Tired it but got the below error :( ######################### Fatal error: Needed to prompt for a connection or sudo password (host: ec2-54-97-244-137.compute-1.amazonaws.com), but input would be ambiguous in parallel mode

Aborting.

Fatal error: Needed to prompt for a connection or sudo password (host: ec2-54-21-201-182.compute-1.amazonaws.com), but input would be ambiguous in parallel mode

Aborting.

Fatal error: Needed to prompt for a connection or sudo password (host: ec2-54-17-231-18.compute-1.amazonaws.com), but input would be ambiguous in parallel mode

Aborting. ##############

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53#issuecomment-139294843 .

cassandrauser commented 9 years ago

Do we need to enable passwordless login for all the nodes?. And for which user do we need to enable?.

tbarbugli commented 9 years ago

the error is not about login but about sudo password prompt. the user is allowed to connect to your Cassandra nodes but not to use sudo without password prompt. you should either use a user that does not get prompted for a password or adjust your sudo configs (visudo)

On Thu, Sep 10, 2015 at 6:41 PM cassandrauser notifications@github.com wrote:

Do we need to enable passwordless login for all the nodes?. And for which user do we need to enable?.

— Reply to this email directly or view it on GitHub https://github.com/tbarbugli/cassandra_snapshotter/issues/53#issuecomment-139305767 .