Open CTTY opened 7 months ago
@CTTY I was trying to reproduce this issue, but got into some other setup issue. Will get back to you soon on this.
@ad1happy2go Getting the similar error while running the commit rollback
, compaction scheduleAndExecute
, compaction schedule
and savepoint create
commit rollback --commit 20240408231846380
24/04/08 23:22:02 INFO InputStreamConsumer: Apr 08, 2024 11:22:02 PM org.apache.spark.launcher.Log4jHotPatchOption staticJavaAgentOption
24/04/08 23:22:02 INFO InputStreamConsumer: WARNING: spark.log4jHotPatch.enabled is set to true, but /usr/share/log4j-cve-2021-44228-hotpatch/jdk17/Log4jHotPatchFat.jar does not exist at the configured location
24/04/08 23:22:02 INFO InputStreamConsumer:
24/04/08 23:22:03 INFO InputStreamConsumer: Error: Failed to load org.apache.hudi.cli.commands.SparkMain: org/apache/hudi/common/engine/HoodieEngineContext
24/04/08 23:22:03 INFO InputStreamConsumer: 24/04/08 23:22:03 INFO ShutdownHookManager: Shutdown hook called
24/04/08 23:22:03 INFO InputStreamConsumer: 24/04/08 23:22:03 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-272bb6ef-f858-42a6-b9d0-9614f1f36371
24/04/08 23:22:03 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from s3://<path>/
24/04/08 23:22:03 INFO HoodieTableConfig: Loading table properties from s3://<path>mansipp_hudi_mor_table_2/.hoodie/hoodie.properties
24/04/08 23:22:03 INFO S3NativeFileSystem: Opening 's3://<path>/.hoodie/hoodie.properties' for reading
24/04/08 23:22:03 INFO HoodieTableMetaClient: Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3://<path>/mansipp_hudi_mor_table_2/
Commit 20240408231846380 failed to roll back
compaction schedule —hoodieConfigs "hoodie.compact.inline.max.delta.commits=1"
24/04/08 23:34:34 INFO InputStreamConsumer: Apr 08, 2024 11:34:34 PM org.apache.spark.launcher.Log4jHotPatchOption staticJavaAgentOption
24/04/08 23:34:34 INFO InputStreamConsumer: WARNING: spark.log4jHotPatch.enabled is set to true, but /usr/share/log4j-cve-2021-44228-hotpatch/jdk17/Log4jHotPatchFat.jar does not exist at the configured location
24/04/08 23:34:34 INFO InputStreamConsumer:
24/04/08 23:34:36 INFO InputStreamConsumer: Error: Failed to load org.apache.hudi.cli.commands.SparkMain: org/apache/hudi/common/engine/HoodieEngineContext
24/04/08 23:34:36 INFO InputStreamConsumer: 24/04/08 23:34:36 INFO ShutdownHookManager: Shutdown hook called
24/04/08 23:34:36 INFO InputStreamConsumer: 24/04/08 23:34:36 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-e553d601-6f57-4d2f-9543-da0bee777c41
Failed to run compaction for 20240408233433912
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
When using
hudi-cli-with-bundle.sh
to start Hudi CLI, many commands would fail immediately due to error below:Note the error message is also very limited. It seems the stacktrace has been truncated
To Reproduce
Steps to reproduce the behavior:
hudi-cli-with-bundle.sh
connect --path <table_path>
cleans run
Expected behavior
I believe there are 2 problems here:
Environment Description
Hudi version : 0.14.1 (I've tested with both EMR and OSS Hudi 0.14.1)
Spark version : 3.5.0
Hive version : 3.1.3
Hadoop version : 3.3.6
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no
Additional context
Add any other context about the problem here.
Stacktrace
See above