search

Orange-OpenSource / casskop

This Kubernetes operator automates the Cassandra operations such as deploying a new rack aware cluster, adding/removing nodes, configuring the C* and JVM parameters, upgrading JVM and C* versions, and many more...
https://orange-opensource.github.io/casskop/
Apache License 2.0
183 stars 54 forks source link

Backup not getting initiated when cassandra clusters is deployed in different namespace than casskop operator #311

Closed SuhailRahman closed 3 years ago

SuhailRahman commented 3 years ago

Type of question

About general context and help around casskop ?

Question

What did you do? I migrated the operator from namespace-scoped Operator to a cluster-scoped Operator. I deployed the casskop operator under the namespace cassandra-operator-ns and I deployed Cassandra Clusters under the namespace cassandra-ns, as shown below.

kubectl get po -n cassandra-operator-ns
NAME                                          READY   STATUS    RESTARTS   AGE
casskop-cassandra-operator-847bb548d9-7sth9   1/1     Running   0          2d17h
kubectl get po -n cassandra-ns
NAME                          READY   STATUS    RESTARTS   AGE
cassandra-wdc-1-dc1-rack1-0   2/2     Running   0          2d17h
cassandra-wdc-1-dc1-rack2-0   2/2     Running   0          2d17h
cassandra-wdc-1-dc1-rack3-0   2/2     Running   0          2d17h

I created a backup operation for the Cassandra cluster deployed in cassandra-ns namespace.

What did you expect to see? I was expecting to see the backup being initiated and backed up successfully.

What did you see instead? Under which circumstances? I was observing that backup was not getting initiated as the operator and Cassandra cluster are in different namespaces.

$kubectl describe cassandrabackup -n cassandra-ns
Name:         cassandra-wdc-1-backup
Namespace:    cassandra-ns
Labels:       app=cassandra
API Version:  db.orange.com/v1alpha1
Kind:         CassandraBackup
Spec:
  Create Missing Bucket:  true
  Cassandra Cluster:      cassandra-wdc-1
  Datacenter:             dc1
  Schedule:               @every 3m
  Secret:                 cass-backup-secret
  Snapshot Tag:           cassandra-snapshot-1
  Storage Location:       s3://cassandra-wdc-1-cassandra-ns
Status:
Events:
  Type    Reason              Age                       From                        Message
  ----    ------              ----                      ----                        -------
  Normal  BackupNotInitiated  5m29s (x1297 over 2d16h)  cassandrabackup-controller  Backup of datacenter dc1 of cluster cassandra-wdc-1 to s3://cassandra-wdc-1-cassandra-ns under snapshot cassandra-snapshot-1 failed.

Logs of backrest Sidecar

$kubectl logs -n cassandra-ns cassandra-wdc-1-dc1-rack1-0 -c backrest-sidecar
OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
VM settings:
    Max. Heap Size (Estimated): 989.88M
    Using VM: OpenJDK 64-Bit Server VM
12:21:25.184 INFO  com.instaclustr.icarus.Icarus - icarus version: instaclustr-icarus 1.0.10-SNAPSHOT, Build time: , Git commit:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/lib/icarus/icarus-1.0.10-SNAPSHOT.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
12:21:29.569 INFO  com.instaclustr.kubernetes.KubernetesApiModule$InClusterCoreV1ApiProvider - In-cluster CoreV1Api initialized ...
12:21:30.281 INFO  com.instaclustr.guice.Application - Services to start: [UploadTracker [NEW], DownloadTracker [NEW], JerseyHttpServerService [NEW], OperationsService [NEW], OperationsExpirationService [NEW]]
12:21:30.282 INFO  com.instaclustr.guice.Application - Starting services.
12:21:30.284 INFO  com.instaclustr.esop.impl.AbstractTracker - Starting tracker ...
12:21:30.288 INFO  com.instaclustr.esop.impl.AbstractTracker - Starting tracker ...
Mar 26, 2021 12:21:37 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: Parameter 1 of type java.util.Set<java.lang.Class<? extends com.instaclustr.operations.Operation>> from public java.util.Collection<com.instaclustr.operations.Operation> com.instaclustr.sidecar.operations.OperationsResource.getOperations(java.util.Set<java.lang.Class<? extends com.instaclustr.operations.Operation>>,java.util.Set<com.instaclustr.operations.Operation$State>) is not resolvable to a concrete type.

12:21:37.871 INFO  com.instaclustr.sidecar.http.JerseyHttpServerService - Starting HTTP server on address 0.0.0.0/0.0.0.0:4567
12:21:37.986 INFO  com.instaclustr.sidecar.http.JerseyHttpServerService - Started HTTP server on http://[::]:4567/
12:21:38.062 INFO  com.instaclustr.guice.Application - Successfully started all services.

Log of casskop operator

$kubectl logs -n cassandra-operator-ns casskop-cassandra-operator-847bb548d9-7sth9
time="2021-03-29T05:16:44Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:16:44Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:44Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:16:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:16:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:16:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:16:49Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:16:49Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:49Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:16:49Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:49Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:16:49Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:49Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:16:49Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:49Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:16:54Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:16:54Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:54Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:16:54Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:54Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:16:54Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:54Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:16:54Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:54Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:16:59Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:16:59Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:59Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:16:59Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:59Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:16:59Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:59Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:16:59Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:16:59Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:04Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:04Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:04Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:04Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:04Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:04Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:04Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:04Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:04Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:09Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:09Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:09Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:09Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:09Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:09Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:09Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:09Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:09Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:14Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:14Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:14Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:14Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:14Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:14Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:14Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:14Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:14Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:19Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:19Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:19Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:19Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:19Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:19Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:19Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:19Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:19Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:24Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:24Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:24Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:24Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:24Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:24Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:24Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:24Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:24Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:29Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:29Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:29Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:29Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:29Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:29Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:29Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:29Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:29Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:31Z" level=info msg="Retrieving Cassandra Sidecar client for cassandra-ns/cassandra-wdc-1"
time="2021-03-29T05:17:31Z" level=error msg="Post \"http://cassandra-wdc-1-dc1-rack2-0.cassandra-wdc-1:4567/operations\": dial tcp: lookup cassandra-wdc-1-dc1-rack2-0.cassandra-wdc-1 on 10.96.0.10:53: no such hostError while starting backup operation" Request.Name=cassandra-wdc-1-backup Request.Namespace=cassandra-ns
time="2021-03-29T05:17:34Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:34Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:34Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:34Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:34Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:34Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:34Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:34Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:34Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:39Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:39Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:39Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:39Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:39Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:39Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:39Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:39Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:39Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:44Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:44Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:44Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:49Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:49Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:49Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:49Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:49Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:49Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:49Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:49Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:49Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:54Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:54Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:54Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:54Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:54Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:54Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:54Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:54Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:54Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:17:59Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:17:59Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:59Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:17:59Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:59Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:17:59Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:59Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:17:59Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:17:59Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:04Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:04Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:04Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:04Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:04Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:04Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:04Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:04Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:04Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:09Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:09Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:09Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:09Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:09Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:09Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:09Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:09Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:09Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:14Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:14Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:14Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:14Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:14Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:14Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:14Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:14Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:14Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:19Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:19Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:19Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:19Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:19Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:19Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:19Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:19Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:19Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:24Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:24Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:24Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:24Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:24Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:24Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:24Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:24Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:24Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:29Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:29Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:29Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:29Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:29Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:29Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:29Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:29Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:29Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:34Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:34Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:34Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:34Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:34Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:34Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:34Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:34Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:34Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:39Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:39Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:39Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:39Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:39Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:39Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:39Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:39Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:39Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:44Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:44Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:44Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
time="2021-03-29T05:18:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack1 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack1
time="2021-03-29T05:18:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack2 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack2
time="2021-03-29T05:18:44Z" level=error msg="Executing pod operation failed" cluster=cassandra-wdc-1 dc-rack=dc1-rack3 err="Cannot check if there are joining nodes: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:44Z" level=warning msg="Should Not see this message ;) Waiting Rack to be running before continuing, we loop on Next Rack, maybe we don't want that" LastActionName=Initializing LastActionStatus=Done cluster=cassandra-wdc-1 dc-rack=dc1-rack3
time="2021-03-29T05:18:49Z" level=info msg="We will request : cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to catch hostIdMap" cluster=cassandra-wdc-1 err="<nil>"
time="2021-03-29T05:18:49Z" level=error msg="Failed to call cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 to get hostIdMap" cluster=cassandra-wdc-1 err="Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host"
time="2021-03-29T05:18:49Z" level=error msg="CheckPodsState Error: Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/\": dial tcp: lookup cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 on 10.96.0.10:53: no such host" cluster=cassandra-wdc-1
cscetbon commented 3 years ago

@SuhailRahman you have some DNS or network policies issues as far as I can tell 

...
Cannot get host id map: HTTP Request Failed: Post \"http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1:8778/jolokia/
....
lookup cassandra-wdc-1-dc1-rack2-0.cassandra-wdc-1 on 10.96.0.10:53: no such hostError while starting backup operation

The last log shows that it tried to do a backup but it failed to post the query to the backup sidecar

SuhailRahman commented 3 years ago

@cscetbon This works completely fine when both Cassandra clusters and the casskop operator are in same namespace.

cscetbon commented 3 years ago

@SuhailRahman yeah by default casskop watches its own namespace. You need to update WATCH_NAMESPACE in order to get it to look at another namespace. https://github.com/Orange-OpenSource/casskop/blob/e3a6b29ff9223d99f0eb6d39cb5fc34ff96ed5b5/helm/cassandra-operator/templates/deployment.yaml#L52-L55

SuhailRahman commented 3 years ago

I have made those changes. Casskop is able to bring up pods in different namespace but there is a problem when backup operation is triggered.

cscetbon commented 3 years ago

What is the problem ? It should see the CassandraBackup and CassandraRestore objects and trigger operations.

SuhailRahman commented 3 years ago

Sorry for the late reply. Here, I have the Cassandra cluster in namespace cassandra-ns and the operator is present in cassandra-operator-ns. http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 is not reachable for some reason, I wanted to understand what is the significance of cassandra_seeds, does it have any role with respect to backup operation, and is http://cassandra-wdc-1-dc1-rack1-0.cassandra-wdc-1 reachable to other namespaces.

cscetbon commented 3 years ago

@SuhailRahman seeds are not used specifically for a backup but only for new nodes to join an existing cluster. however nodes need to be reachable between namespaces otherwise it won't do anything. Do you have some network policies in place and you forgot to open the port used to connect to the backrest container ?

cscetbon commented 3 years ago

Closing the ticket until there is a way to reproduce the issue.