ksql v5.3.0 headless operation fails to start - java.lang.NullPointerException: null value in entry: ksql.security.extension.class

[saubury]

KSQL server in headless mode (ie., using a --queries-file parameter for ksql-server-start) fails to start with fatal error

ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:62)
java.lang.NullPointerException: null value in entry: ksql.security.extension.class=null

Steps to recreate

• Create topic

  kafka-topics --zookeeper localhost:2181 --create --partitions 1 --replication-factor 1 --topic MYTOPIC

• Create queries file

  cat > headless.ksql << EOF
  create stream MYSTREAM with (kafka_topic='MYTOPIC', value_format='avro');
  EOF

• Stop running (interactive) server

  confluent local stop ksql-server

• Start KSQL Server in headless mode

  /opt/confluent/bin/ksql-server-start /opt/confluent/etc/ksql/ksql-server.properties --queries-file=./headless.ksql

Server fails to start with error

[2019-08-06 07:27:01,242] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:62)
java.lang.NullPointerException: null value in entry: ksql.security.extension.class=null
    at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:32)
    at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:100)
    at com.google.common.collect.RegularImmutableMap.fromEntries(RegularImmutableMap.java:74)
    at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:464)
    at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:437)
    at io.confluent.ksql.rest.server.computation.KafkaConfigStore$KsqlProperties.<init>(KafkaConfigStore.java:119)
    at io.confluent.ksql.rest.server.computation.KafkaConfigStore$KsqlProperties.createFor(KafkaConfigStore.java:127)
    at io.confluent.ksql.rest.server.computation.KafkaConfigStore.<init>(KafkaConfigStore.java:94)
    at io.confluent.ksql.rest.server.computation.KafkaConfigStore.<init>(KafkaConfigStore.java:80)
    at io.confluent.ksql.rest.server.StandaloneExecutorFactory.create(StandaloneExecutorFactory.java:100)
    at io.confluent.ksql.rest.server.StandaloneExecutorFactory.create(StandaloneExecutorFactory.java:53)
    at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:90)
    at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)

Note - same issue occurs when using ksql.queries.file= setting within ksql-server.properties

[apurvam]

This should be fixed by https://github.com/confluentinc/ksql/pull/3147, right @agavra ?

[agavra]

yes it should be fixed by #3147

[saubury]

Thanks. Is there a work-around for the time being (other than down-grading)?

It appears that ksql.security.extension.class is essentially mandatory in headless mode, and null is not a legal value?

[apurvam]

@agavra I believe you have a couple of workarounds you shared on other channels. Can you post them here and close this out?

[agavra]

For ksql.security.extension.class a valid default is io.confluent.ksql.rest.server.security.KsqlDefaultSecurityExtension. Setting this will cause the default partitions and replication factor setting to complain about null values as well. Any value (e.g. 4, 2) is valid as long as you have sufficient kafka brokers setup.

[saubury]

Thanks @agavra - that solution works well; thank-you!