使用spark查询1.3亿数据 executor 报 OOM

lihuigang / hive-bitmap-udf

在hive中使用Roaring64Bitmap实现精确去重功能

Apache License 2.0

67 stars 33 forks source link

Open 1095661190 opened 1 year ago

1095661190 commented 1 year ago

spark executor 4G sql语句 select default.bitmap_count(default.to_bitmap(id)) as user_num from table

1095661190 commented 1 year ago

lihuigang commented 1 year ago

这个把 spark executor 内存调大就可以了，处理1.3亿的数据压力还是有点大