Issue with clustering using Interactive KsqlDB deployment

gmcruz commented 1 year ago

So I have a problem with Clustering KSQLDB. The problem is not with clustering it is really with 2 of the 3 access modes in Interactive ksqlDB deployment mode with k8s deployment scaled above 1. Understanding that the REST API is experimental still (I think) but the Java Client should be 100 percent, right?

I also verified this by standing up a FULL K8s Cluster on a completely separate cloud provider and going over the same steps, the same issue can be recreated.

AN AGGREGATE from a series of streams to table (MESSAGE_COUNT). SELECT COUNT FROM MESSAGE_COUNT WHERE USERID = [UUID]

kubectl scale deployment.apps/ksqldb-deployment --replicas=1

1) KsqlDB CLI - No problem here. SELECT COUNT FROM MESSAGE_COUNT... 2) KsqlDB Java Client - No problem here. Running a select connecting to the K8s ksqldb service.

No problem here. Running a select connecting to the K8s POD. 3) KsqlDB REST call - No problem here. Run a curl -X "POST" "http://[IP_K8S_KSQLDB_SERVICE]:8088/query-stream"
- No problem here. Run a curl -X "POST" "http://[IP_K8S_POD]:8088/query-stream"

kubectl scale deployment.apps/ksqldb-deployment --replicas=2

I check here that the correct topics have the correct consumers added and this new POD has joined the cluster correctly.

1) KsqlDB CLI

SOME PROBLEM. SELECT COUNT FROM MESSAGE_COUNT... Correct count is returned, but a message is also sent back with good result - "io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to try.Forwarding pull query request failed with error Error issuing POST to KSQL server. path:/query"

2) KsqlDB Java Client
PROBLEM HERE. Running a select connecting to the K8s ksqldb service. message is always sent back - java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not decode JSON, expected consistency token: {"errorMessage":{"@type":"generic_error","error_code":50000,"message":"io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to try.Forwarding pull query request failed with error Error issuing POST to KSQL server. path:/query "}}
PROBLEM HERE. Running a select connecting to the K8s POD. message is always sent back - java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could not decode JSON, expected consistency token:
{"errorMessage":{"@type":"generic_error","error_code":50000,"message":"io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to
try.Forwarding pull query request failed with error Error issuing
POST to KSQL server. path:/query "}}

3) KsqlDB REST call
PROBLEM HERE. Run a curl -X "POST" "http://[IP_K8S_KSQLDB_SERVICE]:8088/query-stream" Half the requests come back with no result but a message is always sent back with good result - "io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to try.Forwarding pull query request failed with error Error issuing POST to KSQL server. path:/query"
SOME PROBLEM. Run a curl -X "POST" "http://localhost:8088/query-stream" but a message is also sent back with good result - {"@type":"generic_error","error_code":50000,"message":"io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to try.Forwarding pull query request failed with error Error issuing POST to KSQL server. path:/query "}
PROBLEM HERE TO ORIGINAL SCALE 1 POD. Run a curl -X "POST" "http://localhost:8088/query-stream" but a message is also sent back - {"@type":"generic_error","error_code":50000,"message":"io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to try.Forwarding pull query request failed with error Error issuing POST to KSQL server. path:/query "}
SOME PROBLEM. Run a curl -X "POST" "http://[IP_K8S_POD]:8088/query-stream" but a message is also sent back with good result - {"@type":"generic_error","error_code":50000,"message":"io.confluent.ksql.execution.streams.materialization.MaterializationException: Unable to execute pull query: Exhausted standby hosts to try.Forwarding pull query request failed with error Error issuing POST to KSQL server. path:/query "}

kubectl scale deployment.apps/ksqldb-deployment --replicas=1

Once I scale back to 1 within a few seconds all goes back to working as above with replica 1. There is no doubt that something is going on here. I have done this experiment multiple times, recreating the issue every time.

baliberdin commented 1 year ago

@gmcruz I saw a similar problem and figured it was a misconfiguration of ksql.advertised.listener. When ksql is run as a cluster, each node needs to expose a reachable address. I don't know if this is it, but I think it's worth a try

CrazedCoderNate commented 1 year ago

Having the same issue. Any chance on an update with this? Scaling manually 5 pods using KSQL, seems to be throwing issue on simultaneous queries to the same KSQL table.

gmcruz commented 1 year ago

Having the same issue. Any chance on an update with this? Scaling manually 5 pods using KSQL, seems to be throwing issue on simultaneous queries to the same KSQL table.

Truth is I gave up trying to figure it out. I spent sSO much time on it. I instead dropped KSQLDB in favor of KafkaStreams app. Based on this tutorial: medium.com /bakdata/queryable-kafka-topics-with-kafka-streams-8d2cca9de33f

Sorry I know its not what you want to here but for me it was worth the change. hope it helps.

confluentinc / ksql