search

DarioBalinzo / kafka-connect-elasticsearch-source

Kafka Connect Elasticsearch Source
Apache License 2.0
65 stars 38 forks source link

using filter.whitelist for nested value #72

Open fadlymutil opened 2 years ago

fadlymutil commented 2 years ago

Hi There,

i am having problem when using configuration filters.whitelist for nested value,

my json document value part ;

"resPayload": [
  "{ \"timestamp\":1611882445, \"status\":200, \"errors\":{ \"identity_photo\":\"invalid\" }, \"data\":{ \"name\":true, \"birthdate\":true, \"birthplace\":true, \"address\":\"M*MP*NG\",\"selfie_photo\":80}}"
]

when i use this configuration, i got all value from resPayload field, "filters.whitelist" : "apiName;resPayload",

but when i do this, i only get value apiName field, none from resPayload field "filters.whitelist" : "apiName;resPayload.status",

how do i get only some field value i needed from resPayload

anyone can help me? really appreciate your response.

DarioBalinzo commented 2 years ago

Hi, The problem is that resPayload is not a nested json document, but it is an array of strings.

For this reason, you cannot get resPayload.status

Dario

fadlymutil commented 2 years ago

Hi Dario,

is it still possible to filter resPayload ? can you share how to do it? for now i get all the value of resPayload, and cost large size of messages transfered to kafka, it will be great if i can take out the field i dont needed which has large size (images)

thanks.