search

datastax / cdc-apache-cassandra

Datastax CDC for Apache Cassandra
Apache License 2.0
35 stars 21 forks source link

[Source][UDT] Support optional UDT fields #92

Closed aymkhalil closed 2 years ago

aymkhalil commented 2 years ago

Today, when the C source creates the AVRO schema for the data topic, it assumes all UDT fields are mandatory. Try to pass null (i.e. reading a null column value from C) will cause the Source to break with:

67) [com.datastax.oss-pulsar-functions-instance-2.10.1.3.jar:?]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: java.lang.NullPointerException: null of boolean in field XYZ of ABC of array of union in field XYZ of ABC
        at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:184) ~[java-instance.jar:?]
        at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:178) ~[java-instance.jar:?]
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[java-instance.jar:?]
        at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[java-instance.jar:?]
        at com.datastax.oss.pulsar.source.converters.NativeAvroConverter.serializeAvroGenericRecord(NativeAvroConverter.java:309) ~[?:?]
        at com.datastax.oss.pulsar.source.converters.NativeAvroConverter.toConnectData(NativeAvroConverter.java:301) ~[?:?]
        at com.datastax.oss.pulsar.source.converters.NativeAvroConverter.toConnectData(NativeAvroConverter.java:74) ~[?:?]
        at

It is important to add e2e to cover this use case.