Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
2.46k
stars
966
forks
source link
[Bug] When I submit a task in the sql-client interface for flink to read up paimon to write to oss, there is a crash #4319
Closed
jinmu0410 closed 1 month ago
Search before asking
Paimon version
0.8
Compute Engine
flink 1.18
Minimal reproduce step
CREATE TEMPORARY TABLE messages_oss (
type
string not null comment 'message type, EXTERNAL_IN/EXTERNAL_OUT/INTERNAL',hash
string not null comment 'message hash', src_address string not null comment 'source address, user-friendly address', tx_hash string not null comment 'transaction hash', tx_lt BIGINT not null comment 'transaction lt', workchain INT not null comment 'workchain', shard BIGINT not null comment 'shard', block_seq_no INT not null comment 'block sequence number', dst_address string not null comment 'destination address, user-friendly address', bounce BOOLEAN not null comment 'bounce flag', bounced BOOLEAN not null comment 'bounced flag', amount string not null comment 'amount', ihr_disabled BOOLEAN not null comment 'ihr disabled flag', ihr_fee string not null comment 'ihr fee', fwd_fee string not null comment 'fwd fee', body string not null comment 'message body', body_hash string not null comment 'message body hash', operation_id BIGINT not null comment 'operation id', transfer_comment string not null comment 'transfer comment', state_init_code string not null comment 'state init code', state_init_data string not null comment 'state init data', src_contract string not null comment 'source contract', dst_contract string not null comment 'destination contract', operation_name string not null comment 'operation name', data_json string not null comment 'data json', error string not null comment 'error', created_at timestamp not null comment 'created at', created_lt BIGINT not null comment 'created lt', direction STRING not null COMMENT 'in_msg or out_msg',block_date
STRING )PARTITIONED BY (block_date
) WITH ( 'connector' = 'filesystem', 'path' = 'oss://0xscope-chainbase/chainbase_ton/messages', 'format' = 'parquet' );INSERT INTO messages_oss SELECT
type
,hash
, src_address , tx_hash , tx_lt , workchain , shard , block_seq_no , dst_address , bounce , bounced , amount , ihr_disabled , ihr_fee , fwd_fee , body , body_hash , operation_id , transfer_comment , state_init_code, state_init_data , src_contract , dst_contract , operation_name , data_json , error , created_at , created_lt , direction , DATE_FORMAT(created_at
, 'yyyy-MM-dd') FROM ton.messages /+ OPTIONS('scan.infer-parallelism' = 'false', 'scan.parallelism' = '10') / WHERE created_at >= '2024-01-01' and created_at < '2024-02-01' ;What doesn't meet your expectations?
no
Anything else?
no
Are you willing to submit a PR?