Closed simonsigre closed 3 weeks ago
Can you share the configuration properties for the processor as well?
Fixed with PR
Unsure if this is me or not @david-streamlio but I have pulled down the latest codebase, compiled and appear to be getting errors still... this time for autoUpdatePartitionsInterval
2024-03-06 08:48:59,525 WARN [Timer-Driven Process Thread-7] o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception in Component [PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d]]
java.lang.RuntimeException: Failed to load config into existing configuration data
at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.loadData(ConfigurationDataUtils.java:59)
at org.apache.pulsar.client.impl.ProducerBuilderImpl.loadConf(ProducerBuilderImpl.java:116)
at org.apache.nifi.processors.pulsar.utils.PublisherPool.createLease(PublisherPool.java:73)
at org.apache.nifi.processors.pulsar.utils.PublisherPool.obtainPublisher(PublisherPool.java:57)
at org.apache.nifi.processors.pulsar.pubsub.PublishPulsar.onTrigger(PublishPulsar.java:71)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "autoUpdatePartitionsInterval" (class org.apache.pulsar.client.impl.conf.ProducerConfigurationData), not marked as ignorable (26 known properties: "compressionType", "autoUpdatePartitions", "messageRoutingMode", "lazyStartPartitionedProducers", "maxPendingMessagesAcrossPartitions", "batchingMaxMessages", "chunkingEnabled", "batchingMaxBytes", "batchingPartitionSwitchFrequencyByPublishDelay", "properties", "blockIfQueueFull", "encryptionKeys", "hashingScheme", "topicName", "producerName", "sendTimeoutMs", "initialSequenceId", "accessMode", "initialSubscriptionName", "chunkMaxMessageSize", "batchingEnabled", "multiSchema", "maxPendingMessages", "cryptoFailureAction", "batchingMaxPublishDelayMicros", "autoUpdatePartitionsIntervalSeconds"])
at [Source: (String)"{"autoUpdatePartitionsIntervalSeconds":60,"blockIfQueueFull":false,"batchingMaxBytes":131072,"batchingMaxPublishDelayMicros":1000,"chunkingEnabled":false,"encryptionKeys":[],"lazyStartPartitionedProducers":false,"batchingMaxMessages":1000,"compressionType":"NONE","batchingPartitionSwitchFrequencyByPublishDelay":10,"accessMode":"Shared","maxPendingMessages":0,"hashingScheme":"JavaStringHash","maxPendingMessagesAcrossPartitions":0,"cryptoFailureAction":"FAIL","batchingEnabled":true,"sendTimeoutMs""[truncated 210 chars]; line: 1, column: 665] (through reference chain: org.apache.pulsar.client.impl.conf.ProducerConfigurationData["autoUpdatePartitionsInterval"])
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1132)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2202)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1705)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1683)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:320)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4730)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677)
at org.apache.pulsar.shade.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645)
at org.apache.pulsar.client.impl.conf.ConfigurationDataUtils.loadData(ConfigurationDataUtils.java:57)
... 15 common frames omitted
2024-03-06 08:49:00,134 INFO [NiFi Web Server-121] o.a.n.c.s.StandardProcessScheduler Stopping PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d]
2024-03-06 08:49:00,135 INFO [NiFi Web Server-121] o.a.n.controller.StandardProcessorNode Stopping processor: PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d]
2024-03-06 08:49:00,135 INFO [Timer-Driven Process Thread-3] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PublishPulsar[id=dea1f391-018d-1000-a234-81329227793d] to run
Check the latest version after this PR.
Yes its working fantastic now @david-streamlio thank you so much..
Resolved as part of https://github.com/david-streamlio/pulsar-nifi-bundle/pull/83
Using this as my base (https://central.sonatype.com/artifact/io.streamnative.connectors/nifi-pulsar-nar/overview) and also compiling fresh, the following message is seen on the Nifi processor in 2.0.0-M2 version of Nifi.
Pulsar itself seems to be running fine .. .and the error in Nifi occurs even if the Pulsar server is offline.
Help?