confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
128 stars 1.04k forks source link

`ksql-migrations` tool inserted null for map #9723

Open lihaosky opened 2 years ago

lihaosky commented 2 years ago

Describe the bug When using ksql-migration tool to create table and insert values. null value was inserted for struct type.

To Reproduce Steps to reproduce the behavior, include: There are two files:

  1. V000001__initialize_table_struct_insert_test.sql:

    CREATE TABLE struct_insert_test (
    `id` VARCHAR PRIMARY KEY,
    `testArr` ARRAY<VARCHAR>,
    `testObj` STRUCT<
        `testAttr` VARCHAR
    >
    ) WITH (
    KAFKA_TOPIC = 'struct-insert-test',
    VALUE_FORMAT = 'JSON',
    PARTITIONS = 1
    );
  2. V000002__fill_table_struct_insert_test.sql:

    INSERT INTO STRUCT_INSERT_TEST (
    `id`,
    `testArr`,
    `testObj`
    ) VALUES (
    'TEST_ID',
    ARRAY[
        'TEST_ITEM_1',
        'TEST_ITEM_2'
    ],
    STRUCT(
        `testAttr` := 'TEST_VALUE'
    )
    );

Started ksql server locally and run

Expected behavior key: TEST_ID, value: {"testArr":["TEST_ITEM_1","TEST_ITEM_2"],"testObj":{"testAttr":"TEST_VALUE"}}, partition: 0

Actual behaviour key: TEST_ID, value: {"testArr":["TEST_ITEM_1","TEST_ITEM_2"],"testObj":{"testAttr":null}}, partition: 0

Additional context Bug is here: https://github.com/confluentinc/ksql/blob/master/ksqldb-rest-app/src/main/java/io/confluent/ksql/api/util/ApiSqlValueCoercer.java#L95-L99. The struct on line 95 is a map with key TESTATTR but the coered on line 97 has field with name testaddr. Because of the case problem, https://github.com/confluentinc/ksql/blob/master/ksqldb-rest-app/src/main/java/io/confluent/ksql/api/util/ApiSqlValueCoercer.java#L111 is false and became null.

suhas-satish commented 1 year ago

cc: @lucasbru