StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
8.92k stars 1.79k forks source link

When importing Flink SQL into Starlocks, if the data format is JSON, the default value cannot be triggered, while if the data format is CSV, the default value can be triggered #34983

Closed YangQian-99 closed 1 month ago

YangQian-99 commented 11 months ago

Steps to reproduce the behavior (Required)

  1. CREATE TABLE send_coupon_info ( dt date NOT NULL COMMENT "分区时间,取自drawTime字段", primary_id varchar(50) NOT NULL COMMENT "主键id,上游表名和主键id拼接而成", insert_time datetime NULL DEFAULT CURRENT_TIMESTAMP COMMENT "插入数据库时间" ) ENGINE=OLAP PRIMARY KEY(dt, primary_id) COMMENT "领券记录表" PARTITION BY RANGE(dt) ( PARTITION p20231112 VALUES [("2023-11-12"), ("2023-11-13")), PARTITION p20231113 VALUES [("2023-11-13"), ("2023-11-14")), PARTITION p20231114 VALUES [("2023-11-14"), ("2023-11-15")), PARTITION p20231115 VALUES [("2023-11-15"), ("2023-11-16")), PARTITION p20231116 VALUES [("2023-11-16"), ("2023-11-17"))) DISTRIBUTED BY HASH(primary_id) BUCKETS 1 PROPERTIES ( "replication_num" = "3", "dynamic_partition.enable" = "true", "dynamic_partition.time_unit" = "DAY", "dynamic_partition.time_zone" = "Asia/Shanghai", "dynamic_partition.start" = "-1", "dynamic_partition.end" = "2", "dynamic_partition.prefix" = "p", "dynamic_partition.buckets" = "1", "dynamic_partition.history_partition_num" = "0", "dynamic_partition.replication_num" = "3", "in_memory" = "false", "storage_format" = "V2", "enable_persistent_index" = "true", "compression" = "LZ4" );
  2. Flink SQL DDL and insert statements: CREATE TABLE send_coupon_info ( dt date, primary_id STRING, PRIMARY KEY(dt,primary_id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://', 'load-url' = '', 'database-name' = 'ods', 'table-name' = 'send_coupon_info', 'username' = 'root', 'password' = '', 'sink.semantic'='at-least-once', 'sink.label-prefix'='send_coupon_info', 'sink.properties.format'='json' ); insert into send_coupon_info select TO_DATE(DATE_FORMAT(create_time,'yyyy-MM-dd'),'yyyy-MM-dd'), CONCAT(tablename,'',cast(id as string)),
    from send_coupon_info_source ;
  3. select current_version() mysql> select current_version(); +-------------------+ | current_version() | +-------------------+ | 2.5.13 a3b58a0 | +-------------------+ 1 row in set (0.01 sec)
  4. View insert All values in the time column are null, and it was found that the default current was not triggered TIMESTAMP value
  5. Modify the DDL 'sink. properties. format'='json 'in Flink SQL to' sink. properties. format '='CSV', which triggers the default current_ TIMESTAMP value, which can obtain the normal storage time image
jlj-incom commented 7 months ago

I experience the same thing.

github-actions[bot] commented 1 month ago

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!