search

hpgrahsl / kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector
Apache License 2.0
153 stars 60 forks source link

Use two WriteModel Strategies in one config? #124

Closed kush1509 closed 4 years ago

kush1509 commented 4 years ago

Hi,

I wanted to ask if it is possible to use two WriteModel Strategies in one config. My use-case is that I need to use the ReplaceOneBusinessKeyStrategy to check for my business key and I want to add timestamps as well so I need the UpdateOneTimestampsStrategy. I am not entirely sure if my understanding of these is correct, but I would like to understand why (if so) we cannot chain such strategies.

hpgrahsl commented 4 years ago

Hi,

Thx for reaching out. The original design to allow for only one WriteModelStrategy has been decided on purpose since in general combining/chaining several such strategies often doesn't make sense. But you are right, that for the case you describe it would be reasonable to have both these strategies applied when writing to the DB. That being said, the idea here for now is to implement your own custom strategy and make it usable within the connector. The only inconvenience in this regard is that the configuration doesn't allow you to specify your own one directly - as you could do e.g. for the IdStrategy - which means until this becomes available you would need to fork the repo and adapt it in this regard. But it's rather straight forward and the "merged" strategy you are talking about results from copying together the code from the other two and giving it a creative name :-)

Also you might want to take a look at the official MongoDB connector which is available here: https://github.com/mongodb/mongo-kafka If you want raise a feature request for this there.

hpgrahsl commented 4 years ago

@kush1509 wanted to check back if my reply was in any way helpful for you? just let me know if there is anything unclear. I'm happy to answer :)

kush1509 commented 4 years ago

@hpgrahsl yes thanks for the answer. I did find a strategy that fit my purpose - UpdateOneBusinessKeyTimestampStrategy that does the two things but you're right, I can also write a custom strategy for more complicated use cases. Thanks!

hpgrahsl commented 4 years ago

THX @kush1509 for reporting back. Glad to hear your found a combined strategy serving your needs. I'm closing this then.

bilalesi commented 2 years ago

UpdateOneBusinessKeyTimestampStrategy do not do upsert, do we have an alternative in this moment ?