Sometimes Avro records are encoded using top level record, for example payload, with all the data fields. Currently, these records are imported into a single table column as JSON strings.
This is usual Avro record in Kafka (in JSON representation)
This will map to single column in an Exasol table and imported into VARCHAR column as JSON string. It is still possible to use Exasol JSON functions to extract fields. However, it can cause issues if JSON string exceeds 2M characters which limit of Exasol VARCHAR type.
It would be nice feature to be able to specify the nested fields to be imported as separate columns.
Acceptance Criteria
Added field specification for second level fields.
Unified field specifications for JSON and Avro
Unit and integration tests are added for the feature
Situation
Sometimes Avro records are encoded using top level record, for example
payload
, with all the data fields. Currently, these records are imported into a single table column as JSON strings.This is usual Avro record in Kafka (in JSON representation)
which could map to this Exasol table:
the last two columns are only for metadata keeping.
But unfortunately, sometimes data is stored with one more top level field, as payload.
This will map to single column in an Exasol table and imported into VARCHAR column as JSON string. It is still possible to use Exasol JSON functions to extract fields. However, it can cause issues if JSON string exceeds 2M characters which limit of Exasol VARCHAR type.
It would be nice feature to be able to specify the nested fields to be imported as separate columns.
Acceptance Criteria