memiiso / debezium-server-iceberg

Replicates any database (CDC events) to Apache Iceberg (To Cloud Storage)
Apache License 2.0
174 stars 35 forks source link

case "double"missing from IcebergchangeEvent.java #217

Closed Rishav-Git closed 1 year ago

Rishav-Git commented 1 year ago

case "double" is missing in the master branch on file: IcebergChangeEvent.scala.

Use Case: When using debezium with MySQL database connector, it converts decimal type values to byte type by default. When the property: decimal.handling.mode is set to double, the value should get converted to double type but the type gets converted to string since default case is string. Follow this link for more details: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-decimal-handling-mode:~:text=time%20zone%20information.-,Decimal%20types,-Debezium%20connectors%20handle

We need to add case double to the existing code to fix decimal.handling.mode property.

I have already used this code and tested it, working good for me.

Example:

private static Type.PrimitiveType icebergFieldType(String fieldName, String fieldType) {
    switch (fieldType) {
      case "int8":
      case "int16":
      case "int32": // int 4 bytes
        return Types.IntegerType.get();
      case "int64": // long 8 bytes
        if (TS_MS_FIELDS.contains(fieldName)) {
          return Types.TimestampType.withZone();
        } else {
          return Types.LongType.get();
        }
      case "float8":
      case "float16":
      case "float32": // float is represented in 32 bits,
        return Types.FloatType.get();
      case "double":
      case "float64": // double is represented in 64 bits
        return Types.DoubleType.get();
      case "boolean":
        return Types.BooleanType.get();
      case "string":
        return Types.StringType.get();
      case "uuid":
        return Types.UUIDType.get();
      case "bytes":
        return Types.BinaryType.get();
      default:
        // default to String type
        return Types.StringType.get();
      //throw new RuntimeException("'" + fieldName + "' has "+fieldType+" type, "+fieldType+" not supported!");
    }
  }
Rishav-Git commented 1 year ago

@ismailsimsek let me know If this code can be added to existing code and PR can be raised

ismailsimsek commented 1 year ago

Hi @Rishav-Git definitely can be added, please feel free to open PR. Thank you for reporting it

Rishav-Git commented 1 year ago

Please review the PR @ismailsimsek PR: #218

Rishav-Git commented 1 year ago

PR merged to master