Get NullPointerException when schema topic is not 1 partition

sweat123 commented 6 years ago

[2017-12-15 04:03:21,316] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry) [2017-12-15 04:03:21,320] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain) java.lang.NullPointerException at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:66) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:42) at io.confluent.rest.Application.createServer(Application.java:157) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

Method verifySchemaTopic() throw StoreInitializationException when I used a schema topic with 3 partition;It may seem strange I will get a NullPointerException instead of wrong message.

sweat123 commented 6 years ago

schema registry version 4.0.0;

Method createOrVerifySchemaTopic() in KafkaStore.class#init() will throw StoreInitializationException and it is catched in KafkaSchemaRegistry.class#init(), then throw a SchemaRegistryException and is catched in SchemaRegistryRestApplication.class#setupResources(), finally onShutdown() will be called.
```
try {
  schemaRegistry = new KafkaSchemaRegistry(
      schemaRegistryConfig,
      new SchemaRegistrySerializer()
  );
  schemaRegistry.init();
} catch (SchemaRegistryException e) {
  onShutdown();
  log.error("Error starting the schema registry", e);
  System.exit(1);
}
```

Then we will call schemaRegistry.close() which will call kafkaStore.close() in KafkaSchemaRegistry.class;

@Override
public void onShutdown() {

if (schemaRegistry != null) {
  schemaRegistry.close();
}
...

@Override
public void close() {
log.info("Shutting down schema registry");
kafkaStore.close();
...

In KafkaStore.close() we will call kafkaTopicReader.shutdown() kafkaTopicReader is init after createOrVerifySchemaTopic(), so we will get a NPE. producer is also not init.

@Override
public void close() {
kafkaTopicReader.shutdown();
log.debug("Kafka store reader thread shut down");
producer.close();
log.debug("Kafka store producer shut down");
localStore.close();
log.debug("Kafka store shut down complete");
}

@Override
public void init() throws StoreInitializationException {
...
createOrVerifySchemaTopic();
...
producer = new KafkaProducer<byte[], byte[]>(props);
...
this.kafkaTopicReader =
    new KafkaStoreReaderThread<>(this.bootstrapBrokers, topic, groupId,
                                 this.storeUpdateHandler, serializer, this.localStore,
                                 this.noopKey, this.config);
this.kafkaTopicReader.start();
...

jonbuffington commented 6 years ago

@sweat123 I ran into the same issue where the NPE was masking the underlying issue. My fix was similar to yours except I also added a test for localStore:

...
  if (localStore != null) {
    localStore.close();
    log.debug("Kafka store shut down complete");
  }
}

honnel commented 6 years ago

We also run into this NPE issue. As workaround we created the _schemas-Topic manually with only one partition before startup of schema-registry.

any plans for next release date?

dancarroll67 commented 6 years ago

I am getting this same issue. I created "_schemas" manually, started confluent again, but schema_registry still goes down with the same npe. Will there be an update/ release to fix this?

dancarroll67 commented 6 years ago

Can anyone help with this issue? I am on Mac 10.11.6 and java 1.8.0131-b11. I download Confluent open source (confluent-4.0.0), untar it, and run ./confluent start. I get the error above. Two of my teammates go through the same procedure and do not get the error above.

sweat123 commented 6 years ago

@dancarroll67 May be there is something wrong with your topic configuration. You need set your topic "_schemas" with only 1 partition. Schema Registry will verify your schema topic in KafkaStore.verifySchemaTopic(AdminClient admin).

honnel commented 6 years ago

It's currently scheduled to be in the 4.1 release, which is planned for late February (via @confluentinc Twitter account).

honnel commented 6 years ago

Workaround:

Make sure all instances of schema-registry in cluster are shutdown.

Delete Schema Topic:

kafka-topics --zookeeper localhost:2181 --delete --topic _schemas

Make shure deletion has taken place (can take long time, see https://stackoverflow.com/questions/23976670/when-how-does-a-topic-marked-for-deletion-get-finally-removed)

Create topic with following settings:

kafka-topics --create --zookeeper localhost:2181 --topic _schemas --replication-factor 3 --partitions 1 --config cleanup.policy=compact

tarekabouzeid commented 6 years ago

hi honnel, i tried creating the topic as you suggested but still having same issue

sweat123 commented 6 years ago

@tarekabouzeid Maybe there is something wrong with your configuration. Could you show me your topic configuration?

tarekabouzeid commented 6 years ago

@sweat123 i used the below command line to create the topic: kafka-topics --create --zookeeper localhost:2181 --topic _schemas --replication-factor 3 --partitions 1 --config cleanup.policy=compact

and used the below command to start the schema registry: ./schema-registry-start /etc/schema-registry.properties

and in file schema-registry.properties, below is my configuration: listeners=http://0.0.0.0:8081 kafkastore.connection.url=localhost:2181 kafkastore.topic=_schemas debug=false

running on java 1.8.0_161-b12 OS: RHEL 7.4

sweat123 commented 6 years ago

@tarekabouzeid seems no problem. Could you provide exception information?

tarekabouzeid commented 6 years ago

@sweat123 please check the below

[root@drbdmnsitap1v bin]# ./schema-registry-start /usr/software/confluent-4.0.0/etc/schema-registry/schema-registry.properties [2018-02-27 13:25:57,091] INFO SchemaRegistryConfig values: metric.reporters = [] kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit response.mediatype.default = application/vnd.schemaregistry.v1+json kafkastore.ssl.trustmanager.algorithm = PKIX authentication.realm = ssl.keystore.type = JKS kafkastore.topic = _schemas5 metrics.jmx.prefix = kafka.schema.registry kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1 kafkastore.topic.replication.factor = 3 ssl.truststore.password = [hidden] kafkastore.timeout.ms = 500 host.name = drbdmnsitap1v.lbdc kafkastore.bootstrap.servers = [] schema.registry.zk.namespace = schema_registry kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8 kafkastore.sasl.kerberos.service.name = schema.registry.resource.extension.class = ssl.endpoint.identification.algorithm = compression.enable = false kafkastore.ssl.truststore.type = JKS avro.compatibility.level = backward kafkastore.ssl.protocol = TLS kafkastore.ssl.provider = kafkastore.ssl.truststore.location = response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json] kafkastore.ssl.keystore.type = JKS ssl.truststore.type = JKS kafkastore.ssl.truststore.password = [hidden] access.control.allow.origin = ssl.truststore.location = ssl.keystore.password = [hidden] port = 8081 kafkastore.ssl.keystore.location = master.eligibility = true ssl.client.auth = false kafkastore.ssl.keystore.password = [hidden] kafkastore.security.protocol = PLAINTEXT ssl.trustmanager.algorithm = authentication.method = NONE request.logger.name = io.confluent.rest-utils.requests ssl.key.password = [hidden] kafkastore.zk.session.timeout.ms = 30000 kafkastore.sasl.mechanism = GSSAPI kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05 kafkastore.ssl.key.password = [hidden] zookeeper.set.acl = false schema.registry.inter.instance.protocol = http authentication.roles = [*] metrics.num.samples = 2 ssl.protocol = TLS schema.registry.group.id = schema-registry kafkastore.ssl.keymanager.algorithm = SunX509 kafkastore.connection.url = localhost:2181 debug = true listeners = [http://0.0.0.0:8081] kafkastore.group.id = ssl.provider = ssl.enabled.protocols = [] shutdown.graceful.ms = 1000 ssl.keystore.location = ssl.cipher.suites = [] kafkastore.ssl.endpoint.identification.algorithm = kafkastore.ssl.cipher.suites = access.control.allow.methods = kafkastore.sasl.kerberos.min.time.before.relogin = 60000 ssl.keymanager.algorithm = metrics.sample.window.ms = 30000 kafkastore.init.timeout.ms = 60000 (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:175) [2018-02-27 13:25:57,294] DEBUG Logging to org.slf4j.impl.Log4jLoggerAdapter(org.eclipse.jetty.util.log) via org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:176) [2018-02-27 13:25:57,296] INFO Logging initialized @490ms (org.eclipse.jetty.util.log:186) [2018-02-27 13:25:57,659] DEBUG Kafka store zookeeper client shut down (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:549) [2018-02-27 13:25:57,660] INFO Initializing KafkaStore with broker endpoints: PLAINTEXT://drbddnsitap3v.lbdc:9092,PLAINTEXT://drbddnsitap2v.lbdc:9092,PLAINTEXT://drbddnsitap1v.lbdc:9092 (io.confluent.kafka.schemaregistry.storage.KafkaStore:103) [2018-02-27 13:25:57,890] INFO Validating schemas topic _schemas5 (io.confluent.kafka.schemaregistry.storage.KafkaStore:228) [2018-02-27 13:25:57,903] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:719) [2018-02-27 13:25:57,904] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51) java.lang.NullPointerException at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:66) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:42) at io.confluent.rest.Application.createServer(Application.java:157) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)

sweat123 commented 6 years ago

@tarekabouzeid Your kafkastore.topic is "_schemas5";

"_schemas5" has only 1 partition?
Your kafka version is 1.0.0?
your kafka run in localhost?

tarekabouzeid commented 6 years ago

1- yes, 1 partition with 3 replicas 2- Kafka version is 2.1.0 3- zookeeper is on localhost, but the actual kafka are on different machines.

sweat123 commented 6 years ago

@tarekabouzeid

[2018-02-27 13:25:57,890] INFO Validating schemas topic _schemas5 (io.confluent.kafka.schemaregistry.storage.KafkaStore:228) [2018-02-27 13:25:57,903] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:719)

It seems the configuration topic is not correct; SchemaRegistry will verify schema topic with function KafkaStore.verifySchemaTopic; Maybe you can clone the repo and find what is wrong with your configuration;
I can't found out why you will get NPE with listed information; so sorry!

tarekabouzeid commented 6 years ago

@sweat123 when i shutdown my cloudera zookeeper and kafka brokers, and use this command: confluent start schema-registry all services will work fine, so can you please guide me how to make schema-registry run using my already running zookeeper and kafka brokers?

slepkin commented 6 years ago

I'm surprised that the "official" workaround for this involves deleting the _schemas topic; doesn't that make it impossible to read all AVRO messages that were written using the Registry?

terry19850829 commented 6 years ago

I have the same issue , any idea to fix this ?? I have delete the _schemas topic by zkcli , clean the topic data and recreate. Still have this issue...

charsyam commented 6 years ago

@terry19850829 Did you check kafka version and confluent version? You have to match version of them.

terry19850829 commented 6 years ago

@charsyam I have resolve this issue by cleaning all my kafka data. I've just tested confluent platform for using. Some data should be deteled have not been recorded in zk ?

anderseriksson commented 6 years ago

It can also be a permission issue if you are using Kafka with ACL. When Schema Registry is validating the _schemas topic it need DESCRIBE permission (not only consume/produce) so we resolved this problem by the following

kafka-acls --add --allow-principal User:"CN=schemaregistry" --group="*" --operation All --topic="_schemas" --authorizer-properties zookeeper.connect=zookeeper01:2181

mparthas commented 5 years ago

Confluent 5.0.1 has the same problem when I just use my local mac for installation. I have created _schemas and it still does not work.. Any solution for this ?

rayokota commented 5 years ago

The NPE described in the original problem was fixed in 4.1.x

confluentinc / schema-registry

Get NullPointerException when schema topic is not 1 partition #693