couchbase / kafka-connect-couchbase

Kafka Connect connector for Couchbase Server
https://issues.couchbase.com/projects/KAFKAC
Apache License 2.0
73 stars 72 forks source link

enable path to be sourced from the body similar to the id. #11

Closed dgcaron closed 6 years ago

dgcaron commented 6 years ago

in order to add multiple messages to the same document in a json tree, we need a way to set the path dynamically. this PR enables to source the path from the contents of the message.

if the configured path is a json pointer, the value is searched for in the document and removed from the message. if the value isn't a json pointer, the value is treated as a static path inside a document (current behaviour).

example of dynamic path usage

config - couchbase.subdocument.path: /path
message 1 - { "id":"documentid", "path":"myproperty.value1", "foo":"bar"}
message 2 - { "id":"documentid", "path":"myproperty.value2", "foo":"baz"}

this would result in

{
  "myproperty": 
   {
       "value1":
        {
           "foo":"bar"
        },
        "value2":
        {
           "foo":"baz"
        }
   }
}

whereas a static path

config - couchbase.subdocument.path: myproperty.value1
message 1 - { "id":"documentid",  "foo":"bar"}
message 2 - { "id":"documentid",  "foo":"baz"}

would result in

{
  "myproperty": 
   {
       "value1":
        {
           "foo":"baz"
        }
   }
}
cb-sdk-robot commented 6 years ago

Thanks for the pull request!! To ensure quality review, Couchbase employs a code review system based on Gerrit to manage the workflow of changes in addition to tracking our contributor agreements.

Your changes (commit: 6279567cae72fef0565e698d5464d7819db1f7a8) have been pushed to the Couchbase Review Site: http://review.couchbase.org/98087

::SDKBOT/PR:created

dnault commented 6 years ago

Hi @dgcaron, thanks for the PR. Just wanted to make sure you'd seen the comments in Gerrit. The only thing preventing merge is how the ByteBuf's backing array is accessed in SubDocumentWriter -- the other comments are just suggestions. Thanks, David

cb-sdk-robot commented 6 years ago

Your changes (commit: 82c1625c9cd2887ac86ed3b5f13aa351800b2260) have been pushed to the Couchbase Review Site: http://review.couchbase.org/98087

Note: As your pull request contains multiple commits, we have performed an automatic squash of these commits into a single change-set. If this is not the desired behaviour, please consider submitting a pull request per discreet feature.

::SDKBOT/PR:updated

cb-sdk-robot commented 6 years ago

Your changes (commit: e4082f8cd89aa9e17d535d6f8af06e024f8587d9) have been pushed to the Couchbase Review Site: http://review.couchbase.org/98087

::SDKBOT/PR:updated

cb-sdk-robot commented 6 years ago

This Pull Request has been closed as the associated Gerrit change was merged.

::SDKBOT/PR:merged