confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
119 stars 1.04k forks source link

Ksqldb High Availability - how to reach it #9618

Open mdirusso opened 2 years ago

mdirusso commented 2 years ago

I’m working with a ksqldb server deployed in Kubernetes, and since some time ago it crashed for some reason, I want to implement High Availability as described in https://docs.ksqldb.io/en/latest/operate-and-deploy/high-availability/

We are deploying the server with docker, so the properties that we put inside the config file are:

KSQL_KSQL_STREAMS_NUM_STANDBY_REPLICAS: “2” KSQL_KSQL_QUERY_PULL_ENABLE_STANDBY_READS: “true” KSQL_KSQL_HEARTBEAT_ENABLE: “true” KSQL_KSQL_LAG_REPORTING_ENABLE: “true”

When doing so and restarting the server, I can see that only the first 2 properties are properly set, and I can see the last two (for example with SHOW PROPERTIES from the ksqdb CLI).

Do you have an idea about why I can’t see them?

Do I have to manually deploy a second ksqldb server with the same ksql.service.id? If this is the case, what is the correct way to do it? Are there particular properties to be set?

Thank you very much in advance for the answers.

jaceklaskowski commented 2 years ago

My understanding is like yours that a ksqlDB cluster is ksqlDB instances with the same ksql.service.id. They will use the same command topic and should receive all the queries to execute.

One config that has to be changed though is listeners.

No idea about the properties not set and visible using SHOW PROPERTIES.