Hudi job hangs forever - Githubissues

neerajpadarthi commented 3 months ago

Describe the problem you faced

I encountered an issue using EMR 6.7 with Hudi Version 0.11.0 where the Hudi upsert job did not terminate gracefully and remained stuck in a running state indefinitely, despite the upsert operation being completed. Seems like issues with the post-commit actions but forcefully terminating the job and retrying succeeded. Below provided are the configurations and stacktrace details.

Hudi Configs

Hoodie metadata stats are enabled : spark.conf.set("hoodie.metadata.enable","true")
Writer configs - Map(hoodie.datasource.hive_sync.database -> aws_tst_db, hoodie.parquet.small.file.limit -> 104857600, hoodie.datasource.hive_sync.mode -> hms, hoodie.datasource.hive_sync.support_timestamp -> true, path -> s3://<>/aws_tst_db/o_aws_rates_partitioned_tst, hoodie.datasource.write.precombine.field -> last_update_date, hoodie.datasource.hive_sync.partition_fields -> creation_date_partition, hoodie.datasource.hive_sync.use_jdbc -> false, hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.MultiPartKeysValueExtractor, hoodie.parquet.max.file.size -> 268435456, hoodie.cleaner.parallelism -> 1500, hoodie.parquet.block.size -> 268435456, hoodie.datasource.hive_sync.table -> o_aws_rates_partitioned_tst, hoodie.datasource.meta_sync.condition.sync -> true, hoodie.index.type -> GLOBAL_BLOOM, hoodie.datasource.write.operation -> upsert, hoodie.datasource.hive_sync.enable -> true, hoodie.datasource.write.recordkey.field -> rate_id, hoodie.table.name -> o_aws_rates_partitioned_tst, hoodie.datasource.write.hive_style_partitioning -> true, hoodie.cleaner.policy -> KEEP_LATEST_COMMITS, hoodie.keep.max.commits -> 15, hoodie.upsert.shuffle.parallelism -> 750, hoodie.cleaner.commits.retained -> 10, hoodie.keep.min.commits -> 14, hoodie.datasource.write.partitionpath.field -> creation_date_partition), Append

Stacktrace

Add the stacktrace of the error.

Below is the spark logs

24/06/27 07:28:50 INFO Javalin: Stopping Javalin ...
24/06/27 07:29:50 WARN QueuedThreadPool: QueuedThreadPool[qtp1495243767]@591f9bf7{STOPPING,8<=0<=250,i=232,r=-1,q=0}[NO_TRY] Couldn't stop Thread[qtp1495243767-615,5,main]
24/06/27 07:29:50 ERROR Javalin: Javalin failed to stop gracefully
java.util.concurrent.TimeoutException
    at org.apache.hudi.org.apache.jetty.util.FutureCallback.get(FutureCallback.java:130)
    at org.apache.hudi.org.apache.jetty.util.FutureCallback.get(FutureCallback.java:30)
    at org.apache.hudi.org.apache.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
    at org.apache.hudi.org.apache.jetty.server.Server.doStop(Server.java:447)
    at org.apache.hudi.org.apache.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
    at io.javalin.Javalin.stop(Javalin.java:203)
    at org.apache.hudi.timeline.service.TimelineService.close(TimelineService.java:337)
    at org.apache.hudi.client.embedded.EmbeddedTimelineService.stop(EmbeddedTimelineService.java:137)
    at org.apache.hudi.client.BaseHoodieClient.stopEmbeddedServerView(BaseHoodieClient.java:94)
    at org.apache.hudi.client.BaseHoodieClient.close(BaseHoodieClient.java:86)
    at org.apache.hudi.client.BaseHoodieWriteClient.close(BaseHoodieWriteClient.java:1539)
    at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:712)
    at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:313)
    at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:165)

Below is the exception from executor log

24/06/27 07:16:53 INFO Executor: Running task 5223.0 in stage 26.0 (TID 9982)
24/06/27 07:17:09 ERROR PriorityBasedFileSystemView: Got error running preferred function. Trying secondary
org.apache.hudi.exception.HoodieRemoteException: Read timed out
    at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesFromParams(RemoteHoodieTableFileSystemView.java:241)
    at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestBaseFilesBeforeOrOn(RemoteHoodieTableFileSystemView.java:248)
    at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:99)
    at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestBaseFilesBeforeOrOn(PriorityBasedFileSystemView.java:144)
    at org.apache.hudi.index.HoodieIndexUtils.getLatestBaseFilesForPartition(HoodieIndexUtils.java:69)

Below is the exception from Yarm application log

org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockM
Caused by: java.lang.RuntimeException: org.apache.spark.SparkException: Could not find BlockManagerEndpoint1.
    at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:178)
    at org.apache.spark.rpc.netty.Dispatcher.postRemoteMessage(Dispatcher.scala:136)
    at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:684)

Please check and let me know if you need any additional details. Thanks

ad1happy2go commented 3 months ago

@neerajpadarthi Is it possible to upgrade your hudi version. Or If using 0.11.0 can you disable timeline server.

"hoodie.embed.timeline.server", "false"

neerajpadarthi commented 3 months ago

Hi @ad1happy2go, thanks for checking.

Which version do you recommend upgrading to?
Are there any side effects from disabling the timeline server, "hoodie.embed.timeline.server"? May I know the impact on the ingestion write performance? Also, with the configuration change, would I fall back to the direct market type?

xushiyan commented 3 months ago

@neerajpadarthi suggest upgrade to EMR 7.1 with hudi 0.14 amzn version which is closest to OSS 0.15.

neerajpadarthi commented 3 months ago

Hi @xushiyan, Upgrading to Hudi 0.12 or higher is a significant change for our customers because it does not support redshift integration and also requires us to check the integrations with other services/usecases.

Could you kindly explain the potential effects of disabling the timeline server during writes, as suggested by @ad1happy2go in 0.11? Based on this, we will make the trade-off decision. Kindly let us know. Thanks

ad1happy2go commented 3 months ago

@neerajpadarthi You can go through this blog written by @nsivabalan

https://medium.com/@simpsons/timeline-server-in-apache-hudi-b5be25f85e47

neerajpadarthi commented 3 months ago

Thanks for sharing this link. In our case, the upserts are relatively small, affecting only a few files (10–100's), so from the link's benchmarking details we should be relatively good when using the direct markers. However, for the first bulk insert load, we will enable the timeline server to utilize the timeline batches and prevent any S3 throttling errors.

@ad1happy2go - Is the timeout issue fixed in 0.14V? We will evaluate once the redshift supports 0.14 v. Please let us know. Also, I am observing below difference with the indexing stage before and after the marker change during upserts. Can you please help me understand the runtime difference? I am guessing that the "Load latest base files from all partitions" job finished quickly with direct market as it didn't have the overhead of setting the timeline server, but I see significant time taking with this job, "Obtain key ranges for file slices (range pruning=on)" after disabling the timeline marker server. Can you please help me understand the difference during the range pruning with direct. vs. timeline marker server?

timelineserver marker

direct market

keerthiskating commented 2 months ago

I am facing similar issue. I am using Glue 4.0 with Hudi 0.14. The very first run when the table does not exist in Glue catalog works fine. However, over time, the glue job hangs and never finishes. I see errors similar to below.

"Failure Reason": "Error checking presence of partition meta file for s3a:

Since multiple jobs are writing to same table, I have enabled locks and below is the hudi config I am using

"hudi_options" : {
                'hoodie.table.cdc.enabled':'true',
                'hoodie.table.cdc.supplemental.logging.mode': 'data_before_after',

                'hoodie.datasource.write.recordkey.field': 'uuid',
                'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.ComplexKeyGenerator',

                'hoodie.table.name': "transact_table",
                'hoodie.datasource.write.table.name': "transact_table",
                'hoodie.datasource.hive_sync.table': "transact_table",
                'hoodie.datasource.hive_sync.database': "default",

                'hoodie.datasource.write.partitionpath.field': 'a,b,c',
                'hoodie.datasource.hive_sync.partition_fields': 'a,b,c',
                'hoodie.datasource.write.hive_style_partitioning': 'true',
                'hoodie.datasource.hive_sync.enable': 'true',
                'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',

                'hoodie.datasource.write.storage.type': 'COPY_ON_WRITE',
                'hoodie.datasource.write.operation': 'upsert',
                'hoodie.datasource.write.precombine.field': 'uuid',

                'hoodie.datasource.hive_sync.use_jdbc': 'false',
                'hoodie.datasource.hive_sync.mode': 'hms',
                'hoodie.datasource.hive_sync.support_timestamp': 'true',

                'hoodie.write.concurrency.mode': 'OPTIMISTIC_CONCURRENCY_CONTROL',
                'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider',

                'hoodie.write.lock.dynamodb.table': 'hudi_locks_table',
                'hoodie.write.lock.dynamodb.region': 'us-west-2',
                'hoodie.write.lock.dynamodb.endpoint_url': 'dynamodb.us-west-2.amazonaws.com',
                'hoodie.write.lock.dynamodb.partition_key': "transact_table",
                'hoodie.cleaner.policy.failed.writes':'LAZY',
                'hoodie.keep.min.commits': 10,
                'hoodie.keep.max.commits': 20,
                'hoodie.cleaner.commits.retained': 9
}

keerthiskating commented 2 months ago

I did disable timeline server and still job hangs and doesnt succeed/fail.

ad1happy2go commented 2 months ago

@keerthiskating Do we know which stage is getting stuck, Did you checked the driver logs and spark UI ?

keerthiskating commented 1 month ago

I haven't checked the Spark UI for the job where I was trying to read the metadata table. But for the actual job that is upserting to the hudi table, it is getting stuck at Preparing compaction metadata step.

ad1happy2go commented 1 month ago

@keerthiskating Looks like we are already discussion about this here - https://github.com/apache/hudi/issues/11712

apache / hudi

Hudi job hangs forever #11540