Metastore connection is closed properly

[njalan]

I am using spark to ingest&etl based one apache hudi for more than one years. I suddenly faced that the many spark jobs got stuck after Hive sync completed as below: HiveSyncTool: Sync complete for xxx

But as successful job I also got below log after Hive sync completed:
22/12/12 13:14:59 INFO HiveSyncTool: Sync complete for xxx 22/12/12 13:14:59 INFO metastore: Closed a connection to metastore, current connections: 0

All the stuck job I can see message "HiveSyncTool: Sync complete" but without "Closed a connection to metastore, current connections: 0"

But after I killed all the spark batch and streaming jobs and restart all the jobs ,all the jobs are working fine. I check mysql database and metastore server there is no special errors. Jvm is also looks fine.

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.9.0

• Spark version : 3.0.1

• Hive version : 3.1.2

• Hadoop version : 3.2.2

• Storage (HDFS/S3/GCS..) :S3

• Running on Docker? (yes/no) :no

[xushiyan]

@njalan interesting. when you say "suddenly faced that the many spark jobs got stuck after Hive sync completed" meaning you did not change anything or have any deployments of the application? if so then it seems like some network issue. is this re-occurring consistently afterwards?

[njalan]

It just happened once with one and half years. I also disabled that hive meta sync for spark streaming job since each micro will do hive meta sync but no schema changed. By now it is not happed after that.

[njalan]

@xushiyan I have like around 500 jobs with two metastore servers on two vms, one vm is like 16G for hive metastore. Do you think these two metastore servier is enough?