search

ukwa / ukwa-heritrix

The UKWA Heritrix3 custom modules and Docker builder.
9 stars 7 forks source link

NPE in Kafka handling caused crawl to ceace #30

Closed anjackson closed 5 years ago

anjackson commented 5 years ago

Observed failure in Kafka hook likely due to delay in Kafka coming up...

2019-03-14 16:41:30.549 WARNING thread-498 org.apache.kafka.clients.NetworkClient.warn() [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {fc.crawled=UNKNOWN_TOPIC_OR_PARTITION}
Exception in thread "Thread-15" java.lang.NullPointerException
at uk.bl.wap.crawler.frontier.KafkaUrlReceiver$KafkaConsumerRunner.run(KafkaUrlReceiver.java:350)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
anjackson commented 5 years ago

Ah, no, it did restart, that wasn't the problem!