search

canonical / cassandra-k8s-operator

Apache License 2.0
1 stars 4 forks source link

Unable to connect to Cassandra Connection refused. #27

Closed amanThakral002 closed 3 years ago

amanThakral002 commented 3 years ago

I tried to deploy the Cassandra-operator charm using the following commands

$charmcraft build $juju deploy ./cassandra-k8s.charm --resource cassandra-image=cassandra:latest

Pods are in a running state

root@canonical-poc:/home/ubuntu/aman/try/cassandra-operator# kubectl get pods -n omec -w NAME READY STATUS RESTARTS AGE modeloperator-d9d4d9f5b-7f97g 1/1 Running 0 6h6m cassandra-k8s-0 2/2 Running 0 4h55m

But the connection to Cassandra is getting refused logfile of charm sidecar container cassandra.log

amanThakral002 commented 3 years ago

Adding a few more information.

  1. juju debug-log -> debug.log

  2. juju status --relations statusjuju

mmanciop commented 3 years ago

Hi @amanThakral002 , I suspect the issue is that Cassandra is not up yet. It takes a few minutes for the cluster to be set up, and some of it is due to how the charm is written. It ultimately relies on update_status hook calls to occur, which happen by default, IIRC, every 5 minutes. You can change that behavior across the controller:

juju model-config update-status-hook-interval=30s

Then it would take a couple of minutes for Cassandra to become available.

Let me know if it works for you!

dstathis commented 3 years ago

@amanThakral002 Are you still having issues with this? If so, I can help you get it working. Otherwise, I will close this issue.

amanThakral002 commented 3 years ago

@dstathis, sorry I was working on other stuff for a month. I will try it out today and will update you on the status.

amanThakral002 commented 3 years ago

I am getting same error till yet

ran command :

charmcraft pack
 juju deploy ./cassandra-k8s_ubuntu-20.04-amd64.charm --resource cassandra-image=cassandra:latest

error:

 unit-cassandra-k8s-0: 17:03:06 INFO juju.worker.uniter committing "config-changed" hook
unit-cassandra-k8s-0: 17:03:08 INFO juju.worker.uniter found queued "start" hook
unit-cassandra-k8s-0: 17:03:16 INFO unit.cassandra-k8s/0.juju-log Running legacy hooks/start.
unit-cassandra-k8s-0: 17:03:20 WARNING unit.cassandra-k8s/0.juju-log [control connection] Error connecting to 10.1.39.104:9042:
Traceback (most recent call last):
  File "cassandra/cluster.py", line 3522, in cassandra.cluster.ControlConnection._reconnect_internal
  File "cassandra/cluster.py", line 3544, in cassandra.cluster.ControlConnection._try_connect
  File "cassandra/cluster.py", line 1620, in cassandra.cluster.Cluster.connection_factory
  File "cassandra/connection.py", line 831, in cassandra.connection.Connection.factory
  File "/var/lib/juju/agents/unit-cassandra-k8s-0/charm/venv/cassandra/io/libevreactor.py", line 267, in __init__
    self._connect_socket()
  File "cassandra/connection.py", line 898, in cassandra.connection.Connection._connect_socket
ConnectionRefusedError: [Errno 111] Tried connecting to [('10.1.39.104', 9042)]. Last error: Connection refused
unit-cassandra-k8s-0: 17:03:20 ERROR unit.cassandra-k8s/0.juju-log Control connection failed to connect, shutting down Cluster:
Traceback (most recent call last):
  File "cassandra/cluster.py", line 1690, in cassandra.cluster.Cluster.connect
  File "cassandra/cluster.py", line 3488, in cassandra.cluster.ControlConnection.connect
  File "cassandra/cluster.py", line 3533, in cassandra.cluster.ControlConnection._reconnect_internal
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.1.39.104:9042': ConnectionRefusedError(111, "Tried connecting to [('10.1.39.104', 9042)]. Last error: Connection refused")})
unit-cassandra-k8s-0: 17:03:20 INFO unit.cassandra-k8s/0.juju-log Caught exception <class 'cassandra.cluster.NoHostAvailable'>:('Unable to connect to any servers', {'10.1.39.104:9042': ConnectionRefusedError(111, "Tried connecting to [('10.1.39.104', 9042)]. Last error: Connection refused")})
unit-cassandra-k8s-0: 17:03:28 INFO juju.worker.uniter.operation ran "cassandra-pebble-ready" hook (via hook dispatching script: dispatch)
mmanciop commented 3 years ago

Hi @amanThakral002 , these errors are benign, due to the fact that the library we use to connect to Cassandra from the charm is using the same logger as the charm itself. It looks bad, but it means “Cassandra is not up yet”. We plan to look into ways of fixing it, it causes a lot of confusion and it does mot look clean.

Vaishalinicky commented 3 years ago

Hello @mmanciop ,

is there any update ? issue is still there

mmanciop commented 3 years ago

Hi @Vaishalinicky, @dstathis is working on it ;-)