parse nested fields from cursor field value

DarioBalinzo / kafka-connect-elasticsearch-source

Kafka Connect Elasticsearch Source

Apache License 2.0

65 stars 38 forks source link

parse nested fields from cursor field value #96

Closed Danny02 closed 1 year ago

Danny02 commented 1 year ago

Hi,

I have documents in Elastic with the following structure:

{
  "searchProps": {
    "latestUpdate": "2022-10-25T06:21:10.891414"
   }
  "document": {...}
}

I think that I am able to use your connector with the changes of this PR.

P.S. I think the default cursor field _id used in ElasticRepository is wrong, because you pass the document id in the document Map<String, Object> as es-id.

DarioBalinzo commented 1 year ago

Hi @Danny02 , thanks for the PR. I will check this during the next days!

Danny02 commented 1 year ago

Some updates:

I had to update Mockito (minor version) to make it run with Java > 8. I can not download Java 8 anymore and couldn't run the tests locally otherwise
I noticed that the cursor is also used in other places

Danny02 commented 1 year ago

I also have some problems with the conversion of my data to a Struct. I'm debugging this also.

edit: I discovered the problem and fixed it with #97

DarioBalinzo commented 1 year ago

Do you usually download the relase from this repo or from the confluent hub?

In few days I will draft a new release in this repo, and after that I will ask confluent to update also their hub.