search

couchbase / kafka-connect-couchbase

Kafka Connect connector for Couchbase Server
https://issues.couchbase.com/projects/KAFKAC
Apache License 2.0
70 stars 71 forks source link

Addition of subdocument and N1QL mode to Couchbase Sink Connector #6

Closed dgcaron closed 6 years ago

dgcaron commented 6 years ago

I have added code to allow two new types of operations using the sink connector. You can set the mode using the configuration

couchbase.document.mode=DOCUMENT|SUBDOCUMENT|N1QL

this setting defaults to DOCUMENT and leaves the original behavior as is.

  1. SUBDOCUMENT this setting will make the sink work on subdocuments using the official subdocument API. you can specify the following settings to control the behavior: couchbase.subdocument.mode=UPSERT(default)|ARRAYINSERT|ARRAYPREPEND|ARRAYAPPEND|ARRAYINSERTALL|ARRAYPREPENDALL|ARRAYAPPENDALL|ARRAYADDUNIQUE couchbase.subdocument.path='path in document' couchbase.subdocument.createpath=true(default)|false couchbase.subdocument.createdocument=true(default)|false

  2. N1QL this setting will make the sink work with N1QL, generating UPDATE or UPSERT statements ON KEYS . you can specifiy the following settings to control the behavior: couchbase.n1ql.mode=UPSERT(default)|UPDATE couchbase.subdocument.createdocument=true(default)|false in UPDATE mode, data is added to existing documents (more like a merge) whereas UPSERT replaces the content with the data in the sink record. if the createdocument option is set to true and the update yields 0 changed rows, the message is retried with an upsert statement.

i think there is room for improvement in the retrying of the new commands but i might need some help in doing that correctly with RxJava.

cb-sdk-robot commented 6 years ago

Thanks for the pull request!! To ensure quality review, Couchbase employs a code review system based on Gerrit to manage the workflow of changes in addition to tracking our contributor agreements.

To get this change in and collaborate in code review, please register on Gerrit and accept our CLA. The easiest way to do this is to follow the link below, sign in with your GitHub account and then follow through the steps provided on that page to sign an 'Individual' agreement: http://review.couchbase.org/#/settings/new-agreement.

Keep in mind that the emails we are seeing on the commits are: REDACTION FAILED, did*****ron@pv***om

Note: Please contact us if you have any issues registering with Gerrit! If you have not signed our CLA within 7 days, the Pull Request will be automatically closed.

::SDKBOT/PR:no_cla

dnault commented 6 years ago

Thanks! When you register on Gerrit, please make sure to use the same email address as for this PR: did*****ron@pv***om.

dgcaron commented 6 years ago

Hmm that is not my github account.. My email should be a Google mail account.. I registered on Gerrit, using that email. I can do the same for the one you mentioned above if needed?

dnault commented 6 years ago

Yes, please. Once you do, the cb-sdk-robot will automatically import your PR into Gerrit and we can continue the review there.

avsej commented 6 years ago

You can add your second email (from your commit user.email) using gerrit settings:

image

avsej commented 6 years ago

Also keep in mind that gerrit sees your PR as this: https://patch-diff.githubusercontent.com/raw/couchbase/kafka-connect-couchbase/pull/6.patch I can see that you are trying to submit changes on behalf someone else, them make sure he will also register on gerrit and sign CLA: image

The second issue is that your git config set up incorrectly. Seems like you are using email as your user.name and some garbage (I guess the name of the computer) as user.email image You can use this guide to set it correctly: https://help.github.com/articles/setting-your-username-in-git/

After you fix it, please update your commits with git rebase --interactive and git commit --amend --author "New Author Name <email@address.com>"

dgcaron commented 6 years ago

Sorry about that git config mess, the other email is still me (Didier Caron) but if I need to accept the CLA for that email adress I obviously will. I guess I messed up with using two machines. Private one and my corporate one. I'll correct it asap

cb-sdk-robot commented 6 years ago

Your changes (commit: 2730ec7211d7ca7fbfea8a926c1bcf6c87baf785) have been pushed to the Couchbase Review Site: http://review.couchbase.org/91988

Note: As your pull request contains multiple commits, we have performed an automatic squash of these commits into a single change-set. If this is not the desired behaviour, please consider submitting a pull request per discreet feature.

::SDKBOT/PR:created

dgcaron commented 6 years ago

i have seen the comments in gerrit (but wasn't able to comment there). i will process the remarks asap.

avsej commented 6 years ago

Cool. And please, close this PR, and submit the fixes directly to gerrit (you can do git commit --amend, make sure your commit preserves Change-Id string).

To comment on gerrit, you have to use "Reply" button (visible when you signed in). imagen

cb-sdk-robot commented 6 years ago

Your changes (commit: 2927ab091ea48fee4d998391948e176493bede47) have been pushed to the Couchbase Review Site: http://review.couchbase.org/91988

Note: As your pull request contains multiple commits, we have performed an automatic squash of these commits into a single change-set. If this is not the desired behaviour, please consider submitting a pull request per discreet feature.

::SDKBOT/PR:updated

cb-sdk-robot commented 6 years ago

Your changes (commit: fdf3760b360a8dc7043440c7b92216d3dc27f48b) have been pushed to the Couchbase Review Site: http://review.couchbase.org/91988

Note: As your pull request contains multiple commits, we have performed an automatic squash of these commits into a single change-set. If this is not the desired behaviour, please consider submitting a pull request per discreet feature.

::SDKBOT/PR:updated

cb-sdk-robot commented 6 years ago

This Pull Request has been closed as the associated Gerrit change was merged.

::SDKBOT/PR:merged