gudladona
commented
1 year ago We may have some indicators on what is causing this problem
we have a small file limit of 100MB, it appears that this works well (makes larger files and cleans smaller files) for an average partitions that meets the size requirements.
however, for a partition thats very busy/high volume. it seems like its over bucketing the inserts into many files bec based on avg rec size and the size of new inserts it would always exceed the file size limits and causing it to write to a new file group
example, here is number of file groups written for a single instant(commit) in this partition
aws s3 ls s3://<prefix>/<table>/<tenant>/date=20230316/ | awk -F _ '{print $3}' | sort | uniq -c | sort -nk1 | tail
167 20230316203454183.parquet
168 20230316195218670.parquet
168 20230316201208079.parquet
170 20230316200728433.parquet
175 20230316210557345.parquet
180 20230316130454342.parquet
182 20230316212237421.parquet
211 20230316192405566.parquet
245 20230316210251305.parquet
263 20230316204926437.parquetAs we can see here the shear number of small files in this partition is causing a HUGE json response from the driver there by triggering OOM errors.
we need help in figuring out how to tune this.
nfarah86
nsivabalan
TarunMootala
OOM during a Sync or Async clean operation
ENV:
Hudi version: 0.11.1 Java Version 1.8 Spark Version: 3.1.2 EMR version: 6.4 Clean Policy: KEEP_LATEST_BY_HOURS -- 24 hours(default) Clean Parallelism: 200 (default) Metadata: disabled
We have been experiencing consistent OOM errors when running Hudi delta-streamer job in continuous mode. The oom occurs during the "Generating list of file slices to be cleaned" phase. The image below shows the heap growth during the clean operation. The heap growth particularly happens during 2 API calls from the executors getReplacedFileGroupsBefore and getAllFileGroups on the file system view.
Also, we do have jfr files that contain memory profiles during a failed clean operation, Github does not allow us to attach them to the issue.
We also tried the following setting
hoodie.embed.timeline.server.async: truewhich seems to have reduced the heap usage and the clean succeeds. This seems to happen due to the single threaded nature of the async executor. Using this setting we notice the following heap usage. However, the clean operation is slightly less performant. We will continue to use this setting for now, and post any new observations on this ticket.Flame Graph for the Async clean with async timeline server
Flame Graph for Async clean with sync timeline server
Expected behavior
A clean operation should not fail with OOM
Environment Description
Hudi version : 0.11.1
Spark version : 3.2.1
Hadoop version : 3.2.1
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no
Additional context
Add any other context about the problem here.
Stacktrace