search

david-streamlio / pulsar-nifi-bundle

NiFi Processor for Apache Pulsar
Apache License 2.0
16 stars 19 forks source link

failing 'nifi-pulsar-nar-2.0.0-M2' with error 'UnrecognizedPropertyException' #81

Closed simonsigre closed 3 weeks ago

simonsigre commented 5 months ago

Using this as my base (https://central.sonatype.com/artifact/io.streamnative.connectors/nifi-pulsar-nar/overview) and also compiling fresh, the following message is seen on the Nifi processor in 2.0.0-M2 version of Nifi.

Pulsar itself seems to be running fine .. .and the error in Nifi occurs even if the Pulsar server is offline.

Help?

image

2024-03-01 08:01:55,634 ERROR [Timer-Driven Process Thread-7] o.a.n.p.pulsar.pubsub.PublishPulsar PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d] Processing halted: yielding [1 sec]
    at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.loadData(ConfigurationDataUtils.java:59)
    at org.apache.pulsar.client.impl.ProducerBuilderImpl.loadConf(ProducerBuilderImpl.java:116)
    at org.apache.nifi.processors.pulsar.utils.PublisherPool.createLease(PublisherPool.java:73)
    at org.apache.nifi.processors.pulsar.utils.PublisherPool.obtainPublisher(PublisherPool.java:57)
    at org.apache.nifi.processors.pulsar.pubsub.PublishPulsar.onTrigger(PublishPulsar.java:71)
Caused by: org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "enableBatching" (class org.apache.pulsar.client.impl.conf.ProducerConfigurationData), not marked as ignorable (26 known properties: "compressionType", "autoUpdatePartitions", "messageRoutingMode", "lazyStartPartitionedProducers", "maxPendingMessagesAcrossPartitions", "batchingMaxMessages", "chunkingEnabled", "batchingMaxBytes", "batchingPartitionSwitchFrequencyByPublishDelay", "properties", "blockIfQueueFull", "encryptionKeys", "hashingScheme", "topicName", "producerName", "sendTimeoutMs", "initialSequenceId", "accessMode", "initialSubscriptionName", "chunkMaxMessageSize", "batchingEnabled", "multiSchema", "maxPendingMessages", "cryptoFailureAction", "batchingMaxPublishDelayMicros", "autoUpdatePartitionsIntervalSeconds"])
 at [Source: (String)"{"autoUpdatePartitionsIntervalSeconds":60,"chunkingEnabled":false,"encryptionKeys":[],"batchingMaxMessages":1000,"compressionType":"NONE","accessMode":"Shared","hashingScheme":"JavaStringHash","enableBatching":true,"maxPendingMessagesAcrossPartitions":0,"batchingEnabled":true,"sendTimeoutMs":30000,"chunkMaxMessageSize":-1,"autoUpdatePartitions":false,"autoUpdatePartitionsInterval":60,"batchingMaxPublishDelay":10,"blockIfQueueFull":false,"batchingMaxBytes":131072,"batchingMaxPublishDelayMicros":1"[truncated 200 chars]; line: 1, column: 215] (through reference chain: org.apache.pulsar.client.impl.conf.ProducerConfigurationData["enableBatching"])
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4730)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645)
    at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.loadData(ConfigurationDataUtils.java:57)
david-streamlio commented 5 months ago

Can you share the configuration properties for the processor as well?

david-streamlio commented 5 months ago

Fixed with PR

simonsigre commented 5 months ago

Unsure if this is me or not @david-streamlio but I have pulled down the latest codebase, compiled and appear to be getting errors still... this time for autoUpdatePartitionsInterval

2024-03-06 08:48:59,525 WARN [Timer-Driven Process Thread-7] o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in Component [PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d]]
java.lang.RuntimeException: Failed to load config into existing configuration data
    at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.loadData(ConfigurationDataUtils.java:59)
    at org.apache.pulsar.client.impl.ProducerBuilderImpl.loadConf(ProducerBuilderImpl.java:116)
    at org.apache.nifi.processors.pulsar.utils.PublisherPool.createLease(PublisherPool.java:73)
    at org.apache.nifi.processors.pulsar.utils.PublisherPool.obtainPublisher(PublisherPool.java:57)
    at org.apache.nifi.processors.pulsar.pubsub.PublishPulsar.onTrigger(PublishPulsar.java:71)
    at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
    at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
    at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
    at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
    at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "autoUpdatePartitionsInterval" (class org.apache.pulsar.client.impl.conf.ProducerConfigurationData), not marked as ignorable (26 known properties: "compressionType", "autoUpdatePartitions", "messageRoutingMode", "lazyStartPartitionedProducers", "maxPendingMessagesAcrossPartitions", "batchingMaxMessages", "chunkingEnabled", "batchingMaxBytes", "batchingPartitionSwitchFrequencyByPublishDelay", "properties", "blockIfQueueFull", "encryptionKeys", "hashingScheme", "topicName", "producerName", "sendTimeoutMs", "initialSequenceId", "accessMode", "initialSubscriptionName", "chunkMaxMessageSize", "batchingEnabled", "multiSchema", "maxPendingMessages", "cryptoFailureAction", "batchingMaxPublishDelayMicros", "autoUpdatePartitionsIntervalSeconds"])
 at [Source: (String)"{"autoUpdatePartitionsIntervalSeconds":60,"blockIfQueueFull":false,"batchingMaxBytes":131072,"batchingMaxPublishDelayMicros":1000,"chunkingEnabled":false,"encryptionKeys":[],"lazyStartPartitionedProducers":false,"batchingMaxMessages":1000,"compressionType":"NONE","batchingPartitionSwitchFrequencyByPublishDelay":10,"accessMode":"Shared","maxPendingMessages":0,"hashingScheme":"JavaStringHash","maxPendingMessagesAcrossPartitions":0,"cryptoFailureAction":"FAIL","batchingEnabled":true,"sendTimeoutMs""[truncated 210 chars]; line: 1, column: 665] (through reference chain: org.apache.pulsar.client.impl.conf.ProducerConfigurationData["autoUpdatePartitionsInterval"])
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4730)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677)
    at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645)
    at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.loadData(ConfigurationDataUtils.java:57)
    ... 15 common frames omitted
2024-03-06 08:49:00,134 INFO [NiFi Web Server-121] o.a.n.c.s.StandardProcessScheduler Stopping PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d]
2024-03-06 08:49:00,135 INFO [NiFi Web Server-121] o.a.n.controller.StandardProcessorNode Stopping processor: PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d]
2024-03-06 08:49:00,135 INFO [Timer-Driven Process Thread-3] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d] to run
david-streamlio commented 5 months ago

Check the latest version after this PR.

simonsigre commented 5 months ago

Yes its working fantastic now @david-streamlio thank you so much..

image

simonsigre commented 3 weeks ago

Resolved as part of https://github.com/david-streamlio/pulsar-nifi-bundle/pull/83