confluentinc / bottledwater-pg

Change data capture from PostgreSQL into Kafka
http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/
Apache License 2.0
2 stars 0 forks source link

Extracted timestamp value is negative, which is not allowed. #128

Open machty opened 7 years ago

machty commented 7 years ago

I'm trying to use Kafka-Streams to parse the data I've loaded into Kafka using Postgres and I'm getting

Exception in thread "StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Extracted timestamp value is negative, which is not allowed.
        at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:111)
        at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:117)
        at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:143)
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:415)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)

I found the following link http://docs.confluent.io/3.1.0/streams/faq.html#invalid-timestamp-exception

Am I possibly seeing this error because bottlewater was built for 0.9 and doesn't have proper timestamps for 0.10?

machty commented 7 years ago

FYI I "fixed" this by specifying a TIMESTAMP_EXTRACTOR_CLASS_CONFIG of 'org.apache.kafka.streams.processor.WallclockTimestampExtractor' in the consumer.

That said I'd still like to figure out what could/should change about BW to include timestamp information, otherwise we're limited to at-processing timestamps only (which is what the setting above accomplishes).

mmaia commented 6 years ago

Thanks @machty - this solution worked for me also.

Asiddiki commented 6 years ago

Thanks @machty - Above solution also worked for me.

postelrich commented 6 years ago

Which config file do you add this to and what's the exact line?

machty commented 6 years ago

Honestly I have no idea, never really stuck with this learning Kafka. I'd recommend you check out debezium rather than bottledwater though.

parth22594 commented 5 years ago

Thanks @machty. Above solution worked for me as well.