search

hpgrahsl / kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Apache License 2.0
153 stars 61 forks source link

Removed records are put back again immediately. #122

Closed syalinbas closed 4 years ago

syalinbas commented 4 years ago

I have a connector that upserts records into the collection. However if I delete records connector inserts those deleted records immediately. I tried couple of times removing records and this behavior is consistent. Is it how it supposed to work? I am sure that there's no new messages coming into the topic since I manually feed them. I do not do any restarts either. I just delete records from collection.

... "mongodb.document.id.strategy":"at.grahsl.kafka.connect.mongodb.processor.id.strategy.PartialValueStrategy", "mongodb.key.projection.type":"whitelist", "mongodb.key.projection.list":"deviceId,timestamp", "mongodb.replace.one.strategy":"at.grahsl.kafka.connect.mongodb.writemodel.filter.strategy.ReplaceOneBusinessKeyFilterStrategy" ...

Thanks.

hpgrahsl commented 4 years ago

Hi @zbrsy,

I assume by "deleting records" you mean you delete them in the sink i.e. the mongodb target collection... and then you say it gets re-inserted by the connector right away, without new writes into the kafka source topic?

If what you describe is indeed happening I would be very surprised to say the least :) In fact, I currently cannot think of any way how this could work if I wanted to configure this behaviour on purpose.

Of course, I'm very happy to investigate further, so please if possible share more details -> full configuration, sample data from the topic in question so that I can try to locally reproduce.

syalinbas commented 4 years ago

Hi Hans, Thanks for the reply. I know it is extremely weird and I was very hesitant to report it. I was trying to understand KSQLDB's behavior and making a lot of stop/start. However this did not happen between those starts/stops. I said happened because I could not duplicate this next day but I have deleted records from MongoDB and total count of documents came back to 1288 always. I deleted them at least 5 times and with same result. Unless I lost my mind there must be a reason. My hunch is on KSQLDB. Thanks for confirming that there's no such thing and we can close the ticket.

hpgrahsl commented 4 years ago

The offer is still there to help you further investigate this weird behaviour you reported. I also happen to know a thing or two about ksqlDB as well :)

syalinbas commented 4 years ago

Thanks for the offer. Let me try to duplicate it first, if I can, and then I will definitely let you know.