[Bug] [hive] insert hive table select from paimon oom

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

https://paimon.apache.org/

Apache License 2.0

2.16k stars 855 forks source link

[Bug] [hive] insert hive table select from paimon oom #3342

Open gitfortian opened 2 months ago

gitfortian commented 2 months ago

Search before asking

[X] I searched in the issues and found nothing similar.

Paimon version

0.8

Compute Engine

hive

Minimal reproduce step

insert into hive table select * from paimon table limit 10

What doesn't meet your expectations?

it seems limit push down is invalid

Anything else?

No response

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

wg1026688210 commented 2 months ago

Hi~ @gitfortian . Could you sync the size of your Paimon table and the number of data files in this table,did this occur in hive client and how much memory you have set.

wg1026688210 commented 1 month ago

There is an improvement in #3348

wg1026688210 commented 1 month ago

hi~ @gitfortian you can have a try #3348