DTStack / chunjun

A data integration framework
https://dtstack.github.io/chunjun/
Apache License 2.0
3.98k stars 1.69k forks source link

[Bug] [jdbc] mysql 维表使用lru方式加载的数据为空 #1849

Closed libailin closed 10 months ago

libailin commented 10 months ago

Search before asking

What happened

1、当测试kafka与mysql 维表关系查询时,发现使用lru方式加载mysql的维表数据为空

image

2、在日志中会将mysql lru jdbc 连接字符串里的密码明文输出

image

What you expected to happen

--

How to reproduce

-- {"id":2,"money":293.899778,"datethree":"2020-07-30 10:08:22.123"}
CREATE TABLE source
(
    id        bigint,
    money     decimal(10,3),
    PROCTIME AS PROCTIME()
) WITH (
       'connector' = 'kafka-x'
       ,'topic' = 'test'
       ,'properties.bootstrap.servers' = 'xxx:9092'
       ,'properties.group.id' = 'local_test_mysql_lookup'
       ,'scan.startup.mode' = 'latest-offset'
       ,'format' = 'json'
       ,'json.timestamp-format.standard' = 'SQL'
      );

CREATE TABLE side
(
    id          bigint,
    username    varchar,
    email       varchar,
    PRIMARY KEY (id) NOT ENFORCED
) WITH (
      'connector' = 'mysql-x',
      'url' = 'jdbc:mysql://xxx:3306/flinkx_test?useUnicode=true&characterEncoding=utf-8&useSSL=false&connectTimeout=3000&zeroDateTimeBehavior=convertToNull&useunicode=true&characterEncoding=utf8&useSSL=false&useCursorFetch=true&tinyInt1isBit=false&zeroDateTimeBehavior=convertToNull',
      'table-name' = 'user',
      'username' = 'xxx',
      'password' = 'xxx'

--       ,'lookup.cache-type' = 'ALL' -- 维表缓存类型(NONE、LRU、ALL),默认:LRU
--       ,'lookup.cache-period' = '300000' -- ALL维表每隔多久加载一次数据,默认:3600000毫秒
--       ,'lookup.fetch-size' = '2000' -- ALL维表每次从数据库加载的条数,默认:1000条

      ,'lookup.cache-type' = 'LRU' -- 维表缓存类型(NONE、LRU、ALL),默认:LRU
      ,'lookup.cache.max-rows' = '2000' -- lru维表缓存数据的条数,默认:10000条
      ,'lookup.cache.ttl' = '600000' -- lru维表缓存数据的时间,默认:60000毫秒
      );

CREATE TABLE sink
(
    id          bigint,
    money       decimal,
    username    varchar,
    email       varchar,
    PRIMARY KEY (id) NOT ENFORCED  -- 如果定义了,则根据该字段更新。否则追加
) WITH (
      'connector' = 'stream-x'
      );

CREATE TEMPORARY VIEW view_out AS
select
     u.id as id
    , u.money as money
    , s.username as username
    , s.email as email
from source u
left join side FOR SYSTEM_TIME AS OF u.PROCTIME AS s
on u.id = s.id;

insert into sink select * from view_out;

Anything else

No response

Version

master

Are you willing to submit PR?

Code of Conduct