search

mesosphere-backup / dcos-cassandra-service

DEPRECATED—Open source Apache Cassandra running on DC/OS is now replaced by mesosphere/dcos-commons/frameworks/cassandra. This repository will be deleted at the end of 2017.
Apache License 2.0
116 stars 55 forks source link

Backup to S3 without proper credentials hangs #414

Open gkocur opened 7 years ago

gkocur commented 7 years ago

I was trying to perform backup to S3 bucket, but I didn't provide AWS credentials dcos cassandra backup start --external_location=s3:somebucket —backup_name=backup01

The backup status stuck in IN_PROGRESS state:

{ "id": "075e0950-a869-4fd6-8138-17521e32b330", "name": "Snapshot", "steps": [ { "id": "dc290c90-d7a6-4576-a5bf-ab3dc4f1c4fb", "status": "COMPLETE", "name": "snapshot-node-0", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSnapshotStep: 'snapshot-node-0 [dc290c90-d7a6-4576-a5bf-ab3dc4f1c4fb]' has status: 'COMPLETE'." }, { "id": "8bfc2ab2-7ccd-40c3-a0ea-b9d9f84a01fd", "status": "COMPLETE", "name": "snapshot-node-1", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSnapshotStep: 'snapshot-node-1 [8bfc2ab2-7ccd-40c3-a0ea-b9d9f84a01fd]' has status: 'COMPLETE'." }, { "id": "34437cac-989d-4efd-a739-6611aba22d19", "status": "COMPLETE", "name": "snapshot-node-2", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSnapshotStep: 'snapshot-node-2 [34437cac-989d-4efd-a739-6611aba22d19]' has status: 'COMPLETE'." } ], "status": "COMPLETE" }, { "id": "0dc5bda7-4d81-4692-ac34-da694183c878", "name": "Upload", "steps": [ { "id": "e1f7f292-7bda-4010-a49f-2541d5eea729", "status": "IN_PROGRESS", "name": "upload-node-0", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.UploadBackupStep: 'upload-node-0 [e1f7f292-7bda-4010-a49f-2541d5eea729]' has status: 'IN_PROGRESS'." }, { "id": "175aa0b7-79f5-46fb-b10e-0c9381238c0f", "status": "PENDING", "name": "upload-node-1", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.UploadBackupStep: 'upload-node-1 [175aa0b7-79f5-46fb-b10e-0c9381238c0f]' has status: 'PENDING'." }, { "id": "b3dbefbc-0b74-43b2-83a0-685871c3d0a6", "status": "PENDING", "name": "upload-node-2", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.UploadBackupStep: 'upload-node-2 [b3dbefbc-0b74-43b2-83a0-685871c3d0a6]' has status: 'PENDING'." } ], "status": "IN_PROGRESS" }, { "id": "6900fde9-05c8-42ad-9639-c5564a0b5268", "name": "BackupSchema", "steps": [ { "id": "75190241-8eb0-401d-a1d1-0647a2b43ac9", "status": "PENDING", "name": "backupschema-node-0", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSchemaStep: 'backupschema-node-0 [75190241-8eb0-401d-a1d1-0647a2b43ac9]' has status: 'PENDING'." }, { "id": "7e080380-a310-4ac9-9fb8-bbc1c639ed50", "status": "PENDING", "name": "backupschema-node-1", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSchemaStep: 'backupschema-node-1 [7e080380-a310-4ac9-9fb8-bbc1c639ed50]' has status: 'PENDING'." }, { "id": "70e98253-22b6-4538-aae5-8327752c709f", "status": "PENDING", "name": "backupschema-node-2", "message": "com.mesosphere.dcos.cassandra.scheduler.plan.backup.BackupSchemaStep: 'backupschema-node-2 [70e98253-22b6-4538-aae5-8327752c709f]' has status: 'PENDING'." } ], "status": "PENDING" } ], "errors": [], "status": "IN_PROGRESS" }

I stopped this backup with dcos cassandra backup stop. After stopping it dcos cassandra backup status shows no running/pending backup but upload-node-0 task is still running.

Logs from upload-node-0 task:

ERROR [2017-03-09 11:19:30,488] com.mesosphere.dcos.cassandra.executor.tasks.UploadSnapshot: Upload snapshot failed ! java.lang.NullPointerException: null ! at com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver.getPrefixKey(S3StorageDriver.java:62) ~[cassandra-executor.jar:na] ! at com.mesosphere.dcos.cassandra.executor.backup.S3StorageDriver.upload(S3StorageDriver.java:142) ~[cassandra-executor.jar:na] ! at com.mesosphere.dcos.cassandra.executor.tasks.UploadSnapshot.run(UploadSnapshot.java:82) ~[cassandra-executor.jar:na] ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121] ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] ! at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

triclambert commented 6 years ago

This repo is deprecated and will be archived in one week. Please see the latest version of Cassandra or DSE for DC/OS:

https://docs.mesosphere.com/service-docs/cassandra/ https://docs.mesosphere.com/service-docs/dse/ (enterprise-only)