memiiso / debezium-server-iceberg

Replicates any database (CDC events) to Apache Iceberg (To Cloud Storage)
Apache License 2.0
185 stars 35 forks source link

Detect data type in Mongo #323

Open duc-dn opened 4 months ago

duc-dn commented 4 months ago

Hi @ismailsimsek, I am testing cdc the coll with complex schema.

{
  "_id": {
    "$oid": "66431c2ad73dcddc48f55b2a"
  },
  "app_key": "55040710f33156c2ba1872db714b4cf7b0a1a245",
  "device_id": "453E5B45-0B64-4E34-AD17-105725A58631",
  "device_os": "iOS",
  "device_model_name": "iPhone 8",
  "screenX": 375,
  "screenY": 667,
  "selector_events": [
    {
      "instance_class_name": "RCTTextView",
      "vc_class_name": "Settings",
      "orientation": 1,
      "y": 10.5,
      "selector_name": "UNKNOWN",
      "type": "click",
      "height": 17.5,
      "width": 33,
      "x": 6,
      "time": {
        "$numberLong": "1715674110411"
      }
    },
    {
      "orientation": 1,
      "width": 45,
      "height": 17.5,
      "type": "click",
      "instance_class_name": "RCTTextView",
      "y": 14.5,
      "selector_name": "UNKNOWN",
      "time": {
        "$numberLong": "1715674109946"
      },
      "x": 12.5,
      "vc_class_name": "Settings"
    }
  ],
  "multi_touches": [
    {
      "vc_class_name": "Settings",
      "type": "touch",
      "y": 309.5,
      "time": {
        "$numberLong": "1715674110411"
      },
      "x": 141
    },
    {
      "time": {
        "$numberLong": "1715674109946"
      },
      "x": 230.5,
      "y": 313.5,
      "type": "touch",
      "vc_class_name": "Settings"
    },
    {
      "vc_class_name": "Settings",
      "time": {
        "$numberLong": "1715674109396"
      },
      "x": 134,
      "y": 302.5,
      "type": "touch"
    },
    {
      "type": "touch",
      "vc_class_name": "Settings",
      "time": {
        "$numberLong": "1715674108863"
      },
      "y": 303,
      "x": 72
    }
  ],
  "timestamp": {
    "$numberLong": "1715674146818"
  }
}

x, y fields of each array element can be int or double and I got an error:

Check the option 'struct' of parameter 'array.encoding'', error = 'io.debezium.DebeziumException: Field x of schema cdc_.cdc.complex_coll is not the same type for all documents in the array.
ismailsimsek commented 4 months ago

@duc-dn this seems like debezium exception

you could change array encoding parameter