getindata / kafka-connect-iceberg-sink

Apache License 2.0
76 stars 27 forks source link

Inability to Cast Avro Schema to Glue Table Schema with `double` Type #41

Open shahinism opened 1 year ago

shahinism commented 1 year ago

I'm currently experiencing an issue where our connector fails to correctly map an Avro schema to a Glue table schema when using the decimal type. Specifically, for the decimal type, it incorrectly sets the schema of the field to decimal, and the data are inserted as they are in the stored buffer of the original Avro content.

Here are the details of the configuration:

Avro schema:

{
  "fields": [
    {
      "name": "after",
      "type": {
        "name": "after",
        "type": "record",
        "fields": [
          {
            "default": "",
            "name": "OEFRCONUMB",
            "type": {
              "logicalType": "char",
              "maxLength": 2,
              "type": "string"
            }
          },
          {
            "default": "\u0000",
            "name": "OEFRBLNUMB",
            "type": {
              "logicalType": "decimal",
              "precision": 7,
              "scale": 0,
              "type": "bytes"
            }
          }
        ]
      }
    }
  ],
  "name": "header",
  "type": "record"
}

Message body:

data = {
    "after": {
        "OEFRCONUMB": "14",
        "OEFRBLNUMB": Decimal("0"),
    }
}

Here is how the data appears in the Glue table (insert screenshot image here). It seems like the mapping is not functioning as intended.

image

If you could provide any insights or workarounds for this issue, I would greatly appreciate it. Thank you for your assistance.