Kafka Message is junk data

[rcronin]

I'm running the docker compose here and getting absolute junk in the kafka message. Am I doing something wrong here?

[msillence]

It looks like you are viewing the message without avro decoding

All the examples and default config is set for avro, if you use avro for decoding it will probably make more sense

[rcronin]

That definitely got the format of the message looking correct. Still having an issue with the data format for numbers, dates, and timestamps.

[msillence]

so if you check the avro schema it uses the confluent logical types - some things such as kardrop etc don't seem to support them yet Dates are represented as an int days since 1970 but with a logical type

eg.

{
      "name" : "DATE_JOINED",
      "type" : [ "null", {
        "type" : "int",
        "connect.version" : 1,
        "connect.name" : "io.debezium.time.Date"
      } ],
      "default" : null
    },

timestamps by longs:

{
      "name" : "MY_TIMESTAMP",
      "type" : {
        "type" : "long",
        "connect.version" : 1,
        "connect.name" : "io.debezium.time.NanoTimestamp"
      }
    },

[rcronin]

Thanks for looking into this. How should I proceed on this for dates/numbers?

Also to give context on these date/time values, they're not actual date/time data types. It's simply a numeric field on the IBM i.

[rcronin]

Okay I've tracked it down to just decimal values as having problems.

Here's the schema being used.

{
    "name": "FHPRO",
    "type": {
        "type": "bytes",
        "scale": 0,
        "precision": 11,
        "connect.version": 1,
        "connect.parameters": {
            "scale": "0",
            "connect.decimal.precision": "11"
        },
        "connect.name": "org.apache.kafka.connect.data.Decimal",
        "logicalType": "decimal"
    }
}

[msillence]

That looks correct, if you use avro it can decode them natively I use this in our maven pom.xml to generate the correct code from the avro

            <plugin>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-maven-plugin</artifactId>
                <version>1.11.0</version>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>schema</goal>
                        </goals>
                        <configuration>
                            <stringType>String</stringType>
                            <enableDecimalLogicalType>true</enableDecimalLogicalType>
                            <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
                            <outputDirectory>${project.basedir}/target/generated-src/main/java/</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

[msillence]

not a bug