Default migration config with ksql.migrations.topic.replicas=1 fails with NOT_ENOUGH_REPLICAS

[Guilamb]

Describe the bug

With a plain installation of ksqldb from helm chart at https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-ksql-server, when trying to use the migration tool initialize schemas, then it fails on the server side with error traces such as the following

cp-ksql-server-ccc4cf69b-9ff4b cp-ksql-server [2022-09-20 13:34:04,101] WARN [Producer clientId=producer-1] Got error produce response with correlation id 477 on topic-partition cp-ksql-serverksql_MIGRATION_EVENTS-0, retrying (2147483173 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:624)

Modifying ksql-migrations.properties with ksql.migrations.topic.replicas=3 solves the problem

To Reproduce Steps to reproduce the behavior, include:

1. The version of KSQL. 0.28.2
2. Sample source data.
3. Any SQL statements you ran

ksql-migrations apply --config-file ../../data/ksql-migrations.properties --all

Expected behavior

migration tool succeeds with default ksql.migrations.topic.replicas value generated from ksql-migrations new-project command

Actual behaviour A clear and concise description of what actually happens, including:

1. CLI output
2. Error messages
3. KSQL logs

migration tool:

$ ksql-migrations apply --config-file  ../../data/ksql-migrations.properties --all
[...]
io.confluent.ksql.api.client.exception.KsqlClientException: Received error from /inserts-stream. Error code: 50000. Message: Error in processing inserts. Check server logs for details.

Execution time: 121.2620 seconds 

Ksqldb server side (K8S deployment with single deployment):

cp-ksql-server-ccc4cf69b-9ff4b cp-ksql-server [2022-09-20 13:34:04,101] WARN [Producer clientId=producer-1] Got error produce response with correlation id 477 on topic-partition cp-ksql-serverksql_MIGRATION_EVENTS-0, retrying (2147483173 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:624)

Additional context Add any other context about the problem here.

[suhas-satish]

@jzaralim , do you know the answer to this ?

[nick4fake]

It's a bit confusing that default configs and readme doesn't work

[nick4fake]

Is there anyone using ksql in production? What should be done to get basic migrations working?

Edit: getting back to this. After deleting Stream and Table and changing replication factor to 3 it has worked, looks like only factor of 1 gives this issue

[ramirogm]

Hi @nick4fake if you could give a hand:

changing replication factor to 3

Where did you change this? Is this the default.replication.factor = 3 in the cluster? Or something specific to the topic?

[filpano]

Hi @nick4fake if you could give a hand:

changing replication factor to 3

Where did you change this? Is this the default.replication.factor = 3 in the cluster? Or something specific to the topic?

This should be changed in the generated ksql-migrations.properties file that is created when running ksql-migrations new-project <project-name> <ksqldb-url>. it will then be located in ./<project-name>/ksql-migrations.properties which will look as follows:

ksql.server.url=http://ksqldb-primary:8088

# Migrations metadata configs:
# ksql.migrations.stream.name=MIGRATION_EVENTS
# ksql.migrations.table.name=MIGRATION_SCHEMA_VERSIONS
# ksql.migrations.stream.topic.name=ksql-service-idksql_MIGRATION_EVENTS
# ksql.migrations.table.topic.name=ksql-service-idksql_MIGRATION_SCHEMA_VERSIONS
# ksql.migrations.topic.replicas=1

# TLS configs:
# ssl.truststore.location=
# ssl.truststore.password=
# ssl.keystore.location=
# ssl.keystore.password=
# ssl.key.password=
# ssl.key.alias=
# ssl.alpn=false
# ssl.verify.host=true

# ksqlDB server authentication configs:
# ksql.auth.basic.username=
# ksql.auth.basic.password=

# Migrations directory configs:
# ksql.migrations.dir.override=

(output given by 7.6.1) Note the # ksql.migrations.topic.replicas=1 line, which signals that the default number of replicas is 1 (uncomment and change this to n). You need to pass this configuration file to the ksql-migrations tool via the -c <path-to-project>/ksql-migrations.properties.