grassiale
commented
2 years ago Hello, do you have any updates on this issue?
We are currently working around this by manually substituting the mappings on nodes.
What essentially happens is that the nodes of the cassandra cluster (all except one, actually) will have a mapping in /var/lib/cassandra/.restore_mappings that contains ips instead of pod names:
{
"host_map":
{
"10.71.16.133":
{
"seed": false,
"source":
[
"k8ssandra-rack-a-sts-0"
]
},
"10.71.86.153":
{
"seed": false,
"source":
[
"10.71.47.236"
]
},
"localhost":
{
"seed": false,
"source":
[
"10.71.47.233"
]
}
},
"in_place": false
}What we do is, scale the operator to 0 pods, scale the statefulsets to 0, substitute the ip in the localhost key in the json with the respective pod name, on every volume of the cluster and restart the operator. This makes restores work. But is obviously not ideal.
If you think I should open an issue to https://github.com/thelastpickle / cassandra-medusa because it is not inherent to k8ssandra, please let me know.
adejanovski
dnugmanov
Hello again k8ssandra community
What did you do? We are performing a medusa backup of a cassandra datacenter with 3 racks and trying to restore the data on the same cluster. We are not achieving that because medusa-restore is apparently trying to download data from folders that refer to ips of cassandra nodes instead of their pod names. Actually only one of the pods is able to restore the data correctly, the other ones are left by the operator in a podInitCrashloopBackoff state. There are some errors presented below in the operator logs, but they don't tell me much. Fun fact, when I tried the same procedure on a 2 racks cluster without customizations on tokens, it worked. But with the same error in logs.
Did you expect to see some different? We were expecting medusa restore to download data from the same paths on s3 they were uploaded to on all nodes
Environment
K8ssandra Operator version:
v1.1.1Kubernetes version information:Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.12-eks-a64ea69", GitCommit:"d4336843ba36120e9ed1491fddff5f2fec33eb77", GitTreeState:"clean", BuildDate:"2022-05-12T18:29:27Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}Kubernetes cluster kind:``` EKS 1.21Manifests:
Anything else we need to know?: Logs from the container: medusa-restore-logs.log
┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: K8OP-178