[Bug]: streaming: dynamic table do not read source data which is NULL

heni02 commented 11 months ago

Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):57c79f04f35ec80a2e9b95958223c11cec68e13d
- Hardware parameters:
- OS type:
- Others:

Actual Behavior

企业微信截图_65bfcf1d-7134-4f4c-a79a-8d9a75496a60

json: {"c1":"yyjjuejf"} {"c1":NULL}

ddl: create source stream_16(c1 blob )with( "type"='kafka', "topic"= 'test', "partition" = '0', "value"= 'json', "bootstrap.servers"='127.0.0.1:9092'
)

source schema其他类型也有相同的问题，dynamic table无法读取到source null数据

Expected Behavior

No response

Steps to Reproduce

No response

Additional information

No response

gavinyue commented 11 months ago

{"c1":NULL} 不是valid json，所以不能被放到dt.

但是显示在source中，因为它确实是topic里面的一个message.

@LiSong0214 这里需要简单的定一下strategy.

LiSong0214 commented 11 months ago

这个就这样的行为吧。 ●当 kafka message 中的消息为空时，source 查询显示 null，而 dynamic table由于不能 insert null，所以没有这条信息 ●当 kafka message 中的消息与 source 定义的类型全部不匹配，则source 查询显示 null，而 dynamic table由于不能 insert null，所以没有这条信息 ●当 kafka message 中的消息与 source 定义的类型部分不匹配，则 source 和 dynamic table 都保留这条信息，不匹配的置为 null

gavinyue commented 11 months ago

Expected behavior.

gavinyue commented 11 months ago

这个需要和测试确认是否已经解决

gavinyue commented 11 months ago

Pending

gavinyue commented 10 months ago

Pending

gavinyue commented 10 months ago

Pending

gavinyue commented 10 months ago

Pending

gavinyue commented 10 months ago

Pending

qingxinhome commented 10 months ago

We will handle it later

qingxinhome commented 9 months ago

We will handle it later

qingxinhome commented 9 months ago

We will handle it later

qingxinhome commented 9 months ago

Pending processing

qingxinhome commented 9 months ago

Pending processing

qingxinhome commented 9 months ago

Pending processing