search

hpgrahsl / kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Apache License 2.0
153 stars 60 forks source link

Handling of Null values for a field in Sink Record #112

Closed yogi12121993 closed 4 years ago

yogi12121993 commented 4 years ago

How do I avoid update of particular field in mongodb if latest sink document is having null value for that filed? I don't want to update fields with null values. @hpgrahsl Please help me with solution.

hpgrahsl commented 4 years ago

hi @yogi12121993 could you please provide a few more details and maybe a short example of the behaviour you want to achieve in the end? just so that I better understand this "requirement"

yogi12121993 commented 4 years ago

hi @yogi12121993 could you please provide a few more details and maybe a short example of the behaviour you want to achieve in the end? just so that I better understand this "requirement"

For example my sink record is like, "{"c1":"xyz","d1":"abc"}" In this if next record for same unique object id is coming as "{"c1":"xyz","d1":""}" then old value for field d1 will be updated with null right? So how to avoid that?

hpgrahsl commented 4 years ago

ok. thx for clarifying.

1) It's important to differentiate between null and empty string "" :)

2) The exact behaviour how documents are written to MongoDB is defined by chosen WriteModelStrategy and according to the documentation it is the ReplaceOneDefaultStrategy. What's still unclear to me is if you want to only suppress an update of a single field inside the document while updating everything else, or if a specific field is null not updating the whole document at all? Either way, there is currently no configuration that allows you to do this out of the box. You can achieve such very specific behaviour by implementing your own WriteModelStrategy :-) PRs are welcome any time!