search

kubernetes / examples

Kubernetes application example tutorials
Apache License 2.0
6.21k stars 4.54k forks source link

Cassandra: seed provider lists no seeds. #89

Closed Miouge1 closed 6 years ago

Miouge1 commented 7 years ago

Hi,

Following the instruction of the Cassandra example leads to the following error message:

The seed provider lists no seeds.
WARN  14:15:51 Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR 14:15:51 Exception encountered during startup: The seed provider lists no seeds.

It looks like a chicken an egg situation:

If I comment the readinessProbe then the stateful set works.

Shouldn't this use the KubernetesSeedProvider?

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"0480917b552be33e2dba47386e51decb1a211df6", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"269f928217957e7126dc87e6adfa82242bfe5b1e", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

cvieri commented 7 years ago

/sig storage

ekuric commented 7 years ago

I see this issue too, is there any know fix for this?

wallrj commented 6 years ago

You can set PublishNotReadyAddresses = true on the headless service.

See:

publishNotReadyAddresses, when set to true, indicates that DNS implementations must publish the notReadyAddresses of subsets for the Endpoints associated with the Service. The default value is false. The primary use case for setting this field is to use a StatefulSet's Headless Service to propagate SRV records for its Pods without respect to their readiness for purpose of peer discovery. This field will replace the service.alpha.kubernetes.io/tolerate-unready-endpoints when that annotation is deprecated and all clients have been converted to use this field.

Miouge1 commented 6 years ago

@wallrj is that also available as an annotation on the service?

wallrj commented 6 years ago

Yes. My colleague @munnerz notes that the PublishNotReadyAddresses field has only been added in Kubernetes 1.8. Use the annotation service.alpha.kubernetes.io/tolerate-unready-endpoints for backwards compatibility. See:

Vince-Cercury commented 6 years ago

I didn't face the issue but I also noticed that the KubernetesSeedProvider is not enabled in the example. Created a separate issue: https://github.com/kubernetes/examples/issues/147

Eric-Fontana-Bose commented 6 years ago

This just helped me tremendously! Thanks.

pravinsinghal commented 6 years ago

I am running into the same issue

INFO [main] 2018-03-03 09:13:27,994 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility WARN [main] 2018-03-03 09:13:28,091 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local WARN [main] 2018-03-03 09:13:28,094 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-1.cassandra.default.svc.cluster.local The seed provider lists no seeds. Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds. ERROR [main] 2018-03-03 09:13:28,180 CassandraDaemon.java:708 - Exception encountered during startup: The seed provider lists no seeds.

Same issue - Seed provider could not look up cassandra---

Any suggestions how to get past this issue ?

thx.

--pravin

Vince-Cercury commented 6 years ago

@pravinsinghal

Your interpretation of the log is not correct. What is important here is

The seed provider lists no seeds.

There are different ways of specifying seeds with Cassandra. But the bottom line, seed is just a list separated by commas. So you just have to tell cassandra which ones of your nodes are seeds. Your cassandra hasn't even tried to contact a seed node because it's saying there is none defined

By default in the example, the statefulset.yaml specifies the seed node like this:

- name: CASSANDRA_SEEDS value: "cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local"

The next thing that happens is that the run.sh will use this value and replace it in cassandra.yaml at run time.

I'm not sure if you will have time to ssh into the cassandra node before it crashes, but you should check the value in your cassandra.yaml:

`seed_provider:

Addresses of hosts that are deemed contact points.

# Cassandra nodes use this list of hosts to find each other and learn
# the topology of the ring.  You must change this if you are running
# multiple nodes!
#- class_name: io.k8s.cassandra.KubernetesSeedProvider
- class_name: SEED_PROVIDER
  parameters:
      # seeds is actually a comma-delimited list of addresses.
      # Ex: "<ip1>,<ip2>,<ip3>"
      - seeds: "cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local"`

(the default value is 127.0.0.1 in cassanda.yaml but run.sh will replace it. Note that if you want to have more seed you would do:

cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local,cassandra-1.cassandra.%%NAMESPACE%%.svc.cluster.local It's just a comma separated list, nothing else. That's all it takes to specify a node. Cassandra knows what to do next.

pravinsinghal commented 6 years ago

I have defined CASSANDARA_SEEDS env var this is from my yaml file:

env:

And the log says - WARN [main] 2018-03-03 09:13:28,091 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local WARN [main] 2018-03-03 09:13:28,094 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-1.cassandra.default.svc.cluster.local

So - is the lookup failing because the pod crashes before registering the endpoints ? the yaml is a almost a copy cat from default example

Vince-Cercury commented 6 years ago

There are just warning. It would be normal that cassandra-1 is not ready. However for cassandra-0.cassandra.default.svc.cluster.local ...

Are you sure you are using default namespace? You would need to somehow get into the pod before it crashes and see if you can ping cassandra-0.cassandra.default.svc.cluster.local .

pravinsinghal commented 6 years ago

I am using the default namespace.

kubectl get svc --namespace=default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cassandra ClusterIP None 9042/TCP 1d

kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default cassandra-0 0/1 CrashLoopBackOff 487 1d default cassandra-1 0/1 CrashLoopBackOff 487 1d default cassandra-2 0/1 CrashLoopBackOff 488 1d

From kubelet logs: Backing Off restarting container &Container{Name:cassandra,Image:cassandra,Command:[],Args:[],WorkingDir:,Ports:[{intra-node 0 7000 TCP } {tls-intra-node 0 7001 TCP } {jmx 0 7199 TCP } {cql 0 9042 TCP }],Env:[{CASSANDRA_SEEDS cassandra-0.cassandra.default.svc.cluster.local,cassandra-1.cassandra.default.svc.cluster.local nil} {MAX_HEAP_SIZE 256M nil} {HEAP_NEWSIZE 100M nil} {CASSANDRA_CLUSTER_NAME Cassandra nil} {CASSANDRA_DC DC1 nil} {CASSANDRA_RACK Rack1 nil} {POD_IP EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {CASSANDRA_ENDPOINT_SNITCH GossipingPropertyFileSnitch nil}],Resources:ResourceRequirements{Limits:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{1073741824 0} {} 1Gi BinarySI},},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{1073741824 0} {} 1Gi BinarySI},},},VolumeMounts:[{cassandra-data false /var/lib/cassandra/data } {default-token-dcbdx true /var/run/secrets/kubernetes.io/serviceaccount }],LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[IPC_LOCK],Drop:[],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[],TerminationMessagePolicy:File,} in pod cassandra-2_default(1f128c62-1ec3-11e8-a9ca-506b8d4c430a)

pravinsinghal commented 6 years ago

kube dns logs showed this

I0305 07:18:55.550159 1 dns.go:555] Could not find endpoints for service "cassandra" in namespace "default". DNS records will be created once endpoints show up.

Any idea what is happening ?

Vince-Cercury commented 6 years ago

have you create the cassandra service kubectl get svc . What do you get

pravinsinghal commented 6 years ago

Yep the service seems to be there kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cassandra ClusterIP None 9042/TCP 47m

ekuric commented 6 years ago

@pravinsinghal In my case I did below

kubectl create namespace cassandra then inside this project a) created service using .yaml as stated in this repo b) edited cassandra-statefulset.yaml from this repo and changed to have 

- name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"

c) created stateful set kubectl create -f cassandra-statefulset.yaml Also you will need in cassandra-statefulset.yaml to adapt sections related to storageclass at end of that file.

Above worked fine in my case

]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
cassandra-0   1/1       Running   0          7m
cassandra-1   1/1       Running   0          6m
cassandra-2   1/1       Running   0          2m
# kubectl exec cassandra-0 -- nodetool status 
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.21.3.165  99.44 KiB  32           60.4%             a7b45c4a-9c7d-41d8-b253-6b810e5f3972  Rack1-K8Demo
UN  172.20.1.144  99.43 KiB  32           75.6%             2caf06f5-3407-4ae5-9175-ba44d79436a3  Rack1-K8Demo
UN  172.20.3.92   99.43 KiB  32           64.0%             7583ccd4-a3ff-40cb-9342-50cfbc59a614  Rack1-K8Demo
ahmetb commented 6 years ago

cc: @aledbf // this example was recently updated, is this still a problem

pravinsinghal commented 6 years ago

I am still running into this issue - even after i created the namespace.

INFO 17:53:49 Back-pressure is disabled with strategy null. INFO 17:53:49 Unable to load cassandra-topology.properties; compatibility mode disabled WARN 17:53:49 Seed provider couldn't lookup host cassandra-0.cassandra.cassandra.svc.cluster.local WARN 17:53:49 Seed provider couldn't lookup host cassandra-1.cassandra.cassandra.svc.cluster.local Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds. ERROR 17:53:49 Exception encountered during startup: The seed provider lists no seeds

this is the complete yaml

apiVersion: "apps/v1beta1" kind: StatefulSet metadata: name: cassandra namespace: cassandra spec: serviceName: cassandra replicas: 3 updateStrategy: type: RollingUpdate selector: matchLabels: app: cassandra template: metadata: labels: app: cassandra spec: terminationGracePeriodSeconds: 1800 containers:

ekuric commented 6 years ago

@pravinsinghal

if you run cassandra pods in "cassandra" namespace, can you try what I wrote in my previous comment. From your last comment value for CASSANDRA_SEEDS is a bit strange , I guess wrong copy/paste

value: "cassandra-0.cassandra.cassandra.svc.cluster.local,cassandra-1.cassandra.cassandra.svc.cluster.local"

can you try with 

            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"
pravinsinghal commented 6 years ago

Yep - that was a copy/paste issue. I changed that to single cassandra name as recommended - still the same issue

INFO 18:10:53 Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000. INFO 18:10:53 Back-pressure is disabled with strategy null. INFO 18:10:53 Unable to load cassandra-topology.properties; compatibility mode disabled WARN 18:10:53 Seed provider couldn't lookup host cassandra-0.cassandra.cassandra.svc.cluster.local The seed provider lists no seeds. Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds. ERROR 18:10:53 Exception encountered during startup: The seed provider lists no seeds.

aledbf commented 6 years ago

@pravinsinghal please use aledbf/cassandra:v15, it's the test image for #201. You just need to use value: "cassandra-0.cassandra.cassandra.svc.cluster.local" in the env variable

pravinsinghal commented 6 years ago

On a separate cluster created using kubeadm - I was able to successfully deploy this YAML .. 

kubectl exec -ti cassandra-0 --namespace=cassandra -- nodetool status
Datacenter: DC1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.244.1.2  89.88 KiB  32           69.5%             711c4b6d-b5e7-4184-991c-0bbdb1ef79cf  Rack1
UN  10.244.4.2  65.81 KiB  32           63.0%             e12f4027-8a86-41d5-a105-96345393d1b1  Rack1
UN  10.244.2.2  104.55 KiB  32           67.6%             8a7d9f5b-2042-4e9a-910b-f0572c7cc0fb  Rack1

On a separate cluster which is deployed by a separate mechanism (derivative of the hard way model) - this still does not work .. even with 1.9.3. Keep getting "Seed provider couldn't lookup host cassandra-0.cassandra.cassandra.svc.cluster.local".

pravinsinghal commented 6 years ago

Issue got identified. The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.

thx for all the pointers and help.

GervaisYO commented 6 years ago

Hi @pravinsinghal , i'm having the same problem with cassandra on a k8s Cluster with kubeadm. Can you please elabore on how you solve your problem? It was working for me as well before. But then i change my infrastructure. I now have one master and 2 workers. The master node is part of a public subnet and of a private subnet (NAT). I startet the k8s cluster with --apiserver-advertise-address set to the private (NAT Network) ip address of the master node. The workers are part of the NAT only. Previously it was working with every nodes (master and workers) in the NAT network.

Thank you.

ahmetb commented 6 years ago

We recently updated thee cassandra example, ccing @aledbf to take a look and determine if we should close this issue.

aledbf commented 6 years ago

@GervaisYO what image are you using?

GervaisYO commented 6 years ago

I‘m using gcr.io/google-samples/cassandra:v13

fejta-bot commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

madireddyr commented 6 years ago

cc: @aledbf: is this image aledbf/cassandra:v15 is same as the image . cassandra:v14 could you provide docker image file for aledbf/cassandra:v15

fejta-bot commented 6 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

mtbbiker commented 6 years ago

Testing on PS: Kubernetes 1.10.6 in Azure AKS deployment.

Ran into the same problem with gcr.io/google-samples/cassandra:v14 also tried v13. I can get it to work with the default namespace and cassandra service name. But when I tried using a custom namespace ie cassandra the logs complain about no seeds spesified. I have also tried rebuilding the images with env settings other that the default values as spesified in the io.k8s.cassandra.KubernetesSeedProvider README.ms file in the repo path examples/cassandra/java/. I can not confirm but it seems that the KubernetesSeedProvider are unable to overwrite the default settings based on the following log

INFO  11:18:35 Unable to load cassandra-topology.properties; compatibility mode disabled
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
    at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895)
    at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324)
    at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142)
    at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)

So in summary it works with the default namespace but not if you want to deploy in a custom namespace.

slavaaaaaaaaaa commented 6 years ago

Folks, I have this issue in EKS currently. I'm using qa as the namespace and cassandra-service as the service name, which should make this valid:

- name: CASSANDRA_SEEDS
  value: "cassandra-0.cassandra-service.qa.svc.cluster.local"

But at spin-up, neither cassandra-0 nor other containers in the namespace can resolve that hostname (other services resolve fine). Here's the error produced by cassandra-0:

INFO  18:52:07 Back-pressure is disabled with strategy null.
INFO  18:52:07 Unable to load cassandra-topology.properties; compatibility mode disabled
WARN  18:52:07 Seed provider couldn't lookup host cassandra-0.cassandra-service.qa.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR 18:52:07 Exception encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.

I'm using the default suggested service definition (save for publishNotReadyAddresses: true):

apiVersion: v1
kind: Service
metadata:
  name: cassandra-service
  namespace: qa
  labels:
    app: cassandra
spec:
  clusterIP: None
  ports:
    - port: 9042
  publishNotReadyAddresses: true
  selector:
    app: cassandra

Here's the service, running:

$ kubectl get svc --namespace qa
...
cassandra-service    ClusterIP    None    <none>    9042/TCP    3h
...

Here's the pod, failing:

cassandra-0                                   0/1       CrashLoopBackOff   8          19m

I've tried this with the following images, to no avail:

fejta-bot commented 6 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 6 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/examples/issues/89#issuecomment-425669881): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
mkobit commented 5 years ago

Been running into something similar - have been trying to not make the namespace be required as part of the CASSANDRA_SEEDS but haven't gotten it to work - 

        - name: CASSANDRA_SEEDS
          value: "cassandra-0.cassandra.mkobit.svc.cluster.local"
        - name: CASSANDRA_SEED_PROVIDER
          value: "io.k8s.cassandra.KubernetesSeedProvider"

fails for me with

Fatal configuration error; unable to start server.  See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
        at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895)
        at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324)
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142)
        at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
ERROR 18:19:26 Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
        at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647) [apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582) [apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]

This occurs when using image: gcr.io/google-samples/cassandra@sha256:7a3d20afa0a46ed073a5c587b4f37e21fa860e83c60b9c42fec1e1e739d64007. I'd really like to use the namespace without having to hardcode it into the CASSANDRA_SEEDS but not having any luck.

andrew-aernos commented 4 years ago

Fails for me when trying to create the second pod onward. Both the service and statefulsets were applied in the default namespace. I'm playing within a Ubuntu VM's single-node cluster (minikube).

minikube version

v1.6.1

kubectl version

client v1.16.3 server v1.17.0

kubectl get svc cassandra

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cassandra ClusterIP None 9042/TCP 10m

kubectl get pods

NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 7m37s cassandra-1 0/1 CrashLoopBackOff 6 7m

kubectl logs cassandra-1

WARN  23:28:54 Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.
ERROR 23:28:54 Exception encountered during startup: The seed provider lists no seeds.
andrew-aernos commented 4 years ago

Issue got identified. The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.

thx for all the pointers and help.

how might one check the deployed cluster domain?

imran20487 commented 4 years ago

Issue got identified. The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.

thx for all the pointers and help.

This fixed it for me. my-domain was the problem. Changed the domain and it started working.

imran20487 commented 4 years ago

Issue got identified. The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain. thx for all the pointers and help.

how might one check the deployed cluster domain?

I am working in a company which has it's down domain for kube. You may contact your local k8 administrator for the domain.