thelastpickle / cassandra-medusa

Apache Cassandra Backup and Restore Tool
Apache License 2.0
264 stars 142 forks source link

Node hostname differs when invoking backup on commandline vs via gRPC client - K8s #767

Open pvb05 opened 5 months ago

pvb05 commented 5 months ago

Project board link

Hi, I have Medusa 0.21 running in gRPC mode alongside Cassandra in a pod and when I invoke backup directly from the terminal of the Medusa container, the hostname is resolved and it creates and uploads to the corresponding node directory. But when I use a Medusa client to take a backup, it instead uses the node name and does not resolve to the correct hostname.

# LOGS WHEN INVOKED VIA MEDUSA GRPC CLIENT, Node hostname picked up - ip-192-168-125-141

[2024-05-22 23:50:06,658] DEBUG: [S3 Storage] Uploading object from stream -> s3://cassandra-medusa/index/latest_backup/ip-192-168-125-141/backup_name.txt
[2024-05-22 23:50:06,665] INFO: Backup done

# LOGS WHEN INVOKED FROM THE MEDUSA CONTAINER. Node hostname picked up - cassdata-dc1-0

[2024-05-23 00:03:40,645] DEBUG: [S3 Storage] Uploading object from stream -> s3://cassandra-medusa/index/latest_backup/cassdata-dc1-0/backup_name.txt
[2024-05-23 00:03:40,652] INFO: Backup done

I did not set storage.fqdn in medusa.ini hoping it to resolve on its own.

Can someone guide me get this behavior corrected?

thanks, Pradeep

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: MED-8

rzvoncek commented 4 months ago

Hello @pvb05 , this is indeed a strange behaviour.

I tried to reproduce this myself, but with listing the backups. In both cases (medusa list-backups and hitting the GetBackups GRPC endpoint), it resolved into the pod name and I saw a line like:

[2024-06-11 12:14:15,498] DEBUG: Blob foo/foo-dc1-r1-sts-0/backup-2024-05-07t09-15-05-652z/meta/schema.cql was not found in cache.

So there are a few things I'd like to ask you in attempt to narrow this down: