Aiven-Open / opensearch-connector-for-apache-kafka

Aiven's OpenSearch® Connector for Apache Kafka®
Apache License 2.0
63 stars 34 forks source link

logicalType date not correctly indexed #232

Open cb-tomvr opened 10 months ago

cb-tomvr commented 10 months ago

We are using the opensearch connector to sink AVRO records to elasticsearch.

Our AVRO schema looks like this.

{
    "type": "record",
    "name": "Task",
    "fields": [
        {
            "name": "id",
            "type": "string"
        },
        {
            "name": "assigned_to",
            "type": "string"
        },
        {
            "name": "deadline",
            "type": [
                "null",
                {
                    "type": "int",
                    "name": "org.apache.kafka.connect.data.Date",
                    "logicalType": "date"
                }
            ],
            "default": null
        }
    ]
}

The mapping of the deadline property in opensearch is correct

        "deadline": {
          "type": "date"
        }

However, when indexing a record, the days since epoch value is sent to elasticsearch, which is not properly handled by opensearch

{
 ...,
  "_source": {
    "id": "7d9391d6-bc2c-44ce-868a-92095b9ebb01",
    "assigned_to": "953e361e-2507-435e-afcf-4a060c958e97",
    "deadline": 19594,
  },
  "fields": {
    "deadline": [
      "1970-01-01T00:00:19.594Z"
    ]
  }
}