Open BohanZhang0222 opened 1 year ago
There is an option for the sort memory, did you try that, did you try to turn the memory for flink JM?
There is an option for the sort memory, did you try that, did you try to turn the memory for flink JM?
./bin/flink run -ynm compaction_ods_ssp_cloud_vss_on_changed_topic_hudi -yqu eps -ytm 8192 -yd -m yarn-cluster -yt /chj/flink/flink-1.14.0/sql_jar -C file:///chj/flink/flink-1.14.0/sql_jar/juicefs-hadoop-1.0.0-lixiang-dip.jar -c org.apache.hudi.sink.compact.HoodieFlinkCompactor /chj/flink/flink-1.14.0/sql_jar/hudi-flink1.14-bundle-0.13.0.jar --path jfs://XXXX --compaction-max-memory 1024 --seq LIFO --compaction-tasks 10 --min-compaction-interval-seconds 120 --service
FileSystemViewManager Server Run on the host machine instead of JM.It seems the filesystem view takes too much memory. It starts on the client machine that you submit the job.
It seems the filesystem view takes too much memory. It starts on the client machine that you submit the job.
how to adjust client memory ?
Based on how you submit the job, you can config more memory for the start up JVM process.
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
Hudi version : 0.13.0
Flink version : 1.14
Hive version :
Hadoop version : 2.7.6
Storage (HDFS/S3/GCS..) : JuiceFS
Running on Docker? (yes/no) : no
Additional context
Add any other context about the problem here.
Stacktrace
tm log:
compaction service log