search

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.24k stars 2.39k forks source link

[SUPPORT] Flink Streaming read about dynamic day partition #4993

Closed BruceKellan closed 2 years ago

BruceKellan commented 2 years ago

To Reproduce

  1. create a streaming hudi table ods_data, streaming insert into it. ods_data partitioned by day, type.
  2. streaming write into a hudi table dwd_data from ods_data, sql like this: 
    insert into dwd_data
select * from ods_data
where `type` = 'Login'
  3. In next day, dwd_data's max time was '2022-03-08 23:59:59.000'. It seem that it cannot read new data in day=2022-03-09
image

Expected behavior

flink sql + hudi can discover new partition dynamically. Job will auto read new data in day=2022-03-09.

Environment Description

BruceKellan commented 2 years ago

Can someone help me, Can I create partitions in advance?

xushiyan commented 2 years ago

@BruceKellan not very clear what the issue is. what do you mean by this?

In next day, dwd_data's max time was '2022-03-08 23:59:59.000'. It seem that it cannot read new data in day=2022-03-09

BruceKellan commented 2 years ago

@xushiyan

When I start the flink application in day 2022-03-08, the partition day=2022=03-09/type=Login does not exist. The next day (2022-03-09), the partition day=2022=03-09/type=Login will be generated in the table ods_data, The expected behavior is that the flink application can read new data for 2022-03-09, but not now.

BruceKellan commented 2 years ago

I have some idea to optimize and I will open a new issue. Thanks your reply.

weitianpei commented 2 months ago

HI,I encounter this problem again in flink1.16.1-hudi13.1. have you solved this problem？

danny0405 commented 2 months ago

yeah, 0.14.0 supports this fearure.

weitianpei commented 2 months ago

THANKS

weitianpei commented 2 months ago

https://github.com/apache/hudi/issues/11090 I have met a new problem these days,could you please help me.I have post it in github.

---Original--- From: "Danny @.> Date: Thu, Apr 25, 2024 11:26 AM To: @.>; Cc: @.**@.>; Subject: Re: [apache/hudi] [SUPPORT] Flink Streaming read about dynamic daypartition (Issue #4993)

yeah, 0.14.0 supports this fearure.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>