Open kiddom-kq opened 3 years ago
Please add your planning poker estimate with ZenHub @adejanovski
This is very much still an issue, excep we no longer get the CRITICAL
log line. The backup passes through with uploading 0 files.
It'd be nice to print a warning and fail properly instead of pretending all is dandy.
Project board link
I've been working on standing up Medusa the last few days. Most of the issues that I hit have workarounds documented:
But this one didn't; i'm creating this issue to document the problem for future benefit.
On an
admin
node, i would run:Which would then come back with a failure message that appeared to be the same on each node in the cluster
When looking at the
/tmp/medusa-job-<...>/stderr
file on each node, I would see something like this:A bizarre error because just seconds before the error showed up, the
nodetool snapshot
command was executed cluster-wide. Something was wrong with the logic that figured out which stage in the backup process this cluster node was at; the cluster node was behaving like no snapshot had yet been created, but that was obviously not true.Looking closer, it looks like
None
is showing up in the S3 object path.... which indicates something broke and wasn't caught. Dynamic typed languages are wonderful... right up until they're not :/.When looking through the S3 bucket, it was clear that some data was being uploaded. There was a directory structure that Medusa had created and some of the metadata files like
tokenmap.json
had accurate data in them.However, the total size of all objects in the backup path was about
1 MB
which is about a thousand times smaller than the size of data on disk in the cluster.The problem is not in S3 access; Medusa has no problem creating objects in the bucket for internal use so there must be a problem either at the stage where backup files are created or where the files are uploaded. The logs seemed to back this hunch up:
So I started looking through the Medusa codebase to better understand where
nodetool snapshot
files are uploaded to S3.The
CRITICAL: FIND ME:
log is something I added while debugging / tracing the code flow. Thesnapshot.find_dirs()
call returning an empty list explained why the snapshot data files never made it into S3.I spent an embarrassing amount of time stepping through and isolating the code that searches through the configured Cassandra data locations looking for snapshot directories, but eventually figured out why my test code returned the correct set of files and directories while Medusa didn't: permissions.
In short, the medusa backup job was running on the Cassandra node as the
medusa
user. Thecassandra
user owns all directories:If the
medusa
user tries to list the content of/cassandra/data/*/*/snapshots
, thels
operation would fail with some sort ofPermission denied
error.... except this error is silently discarded withglob()
.A quick
sudo chown -R cassandra:medusa
later and backup operations started working:Hopefully that helps somebody! I am slowly making my way through the testing setup for Medusa trying to get a better idea of how it all works so I can contribute some patches to Medusa that will focus on sanity checks / logging with the intent of failing as soon as possible rather than letting silent failures propagate!
┆Issue is synchronized with this Jira Task by Unito ┆friendlyId: K8SSAND-1400 ┆priority: Medium