Trevor-zhang
commented
3 years ago The UFS of our alluxio is HDFS. On Friday, the cluster was expanded horizontally, and data balance was performed after the expansion. At 0:20 on Saturday, we found alluxio hung up.
java.io.IOException: Timed out after waiting 30000 milliseconds for journal entries to be processed
at alluxio.master.journal.raft.RaftJournalWriter.flush(RaftJournalWriter.java:xx2)
at alluxio.master.journal.AsyncJournalWriter.doFlush(AsyncJournalWriter.java:295)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at io.atomix.catalyst.concurrent.BlockingFuture.get(BlockingFuture.java:54)
at alluxio.master.journal.raft.RaftJournalWriter.flush(RaftJournalWriter.java:93)
... 2 more
2020-12-19 00:20:04,843 WARN MasterJournalContext - Journal flush failed. retrying...
java.io.IOException: Timed out after waiting 30000 milliseconds for journal entries to be processed
at alluxio.master.journal.raft.RaftJournalWriter.flush(RaftJournalWriter.java:xx2)
at alluxio.master.journal.AsyncJournalWriter.doFlush(AsyncJournalWriter.java:295)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at io.atomix.catalyst.concurrent.BlockingFuture.get(BlockingFuture.java:54)
at alluxio.master.journal.raft.RaftJournalWriter.flush(RaftJournalWriter.java:93)
... 2 more
2020-12-19 00:20:04,843 WARN MasterJournalContext - Journal flush failed. retrying...
java.io.IOException: Timed out after waiting 30000 milliseconds for journal entries to be processed
at alluxio.master.journal.raft.RaftJournalWriter.flush(RaftJournalWriter.java:xx2)
at alluxio.master.journal.AsyncJournalWriter.doFlush(AsyncJournalWriter.java:295)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at io.atomix.catalyst.concurrent.BlockingFuture.get(BlockingFuture.java:54)
at alluxio.master.journal.raft.RaftJournalWriter.flush(RaftJournalWriter.java:93)
2020-12-19 00:20:24,759 WARN SleepingTimer - Master Active UFS Sync last execution took 34411 ms. Longer than the interval 30000
2020-12-19 00:20:24,759 WARN SleepingTimer - Master Active UFS Sync last execution took 34411 ms. Longer than the interval 30000
2020-12-19 00:27:43,626 ERROR MasterJournalContext - Fatal error: Journal flush failed after 9 attempts
2020-12-19 00:27:43,626 ERROR MasterJournalContext - Fatal error: Journal flush failed after 9 attempts
2020-12-19 00:27:43,626 ERROR MasterJournalContext - Fatal error: Journal flush failed after 9 attempts
2020-12-19 00:27:43,631 INFO BackupLeaderRole - Closing backup-leader role.
2020-12-19 00:27:43,640 INFO AlluxioMasterProcess - Alluxio master ended (lost leadership)
2020-12-19 00:28:20,996 ERROR SingleThreadContext - An uncaught exception occurred
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.ConcurrentSkipListMap.entryIterator(ConcurrentSkipListMap.java:2376)
at java.util.concurrent.ConcurrentSkipListMap$EntrySet.iterator(ConcurrentSkipListMap.java:2522)
at alluxio.master.transport.GrpcMessagingConnection.timeoutPendingRequests(GrpcMessagingConnection.java:489)
at alluxio.master.transport.GrpcMessagingConnection$$Lambda$196/1878270761.run(Unknown Source)
at io.atomix.catalyst.concurrent.Runnables.lambda$logFailure$0(Runnables.java:20)
at io.atomix.catalyst.concurrent.Runnables$$Lambda$96/1541857308.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-12-19 00:28:53,212 INFO BlockWorkerClientPool - Block worker client for GrpcServerAddress{HostName=xx-gray-alhzgz-ecs-dtpbu-data-lake-12, SocketAddress=xx-gray-alhzgz-ecs-dtpbu-data-lake-12/xx.0.74.57:29999} closed.
2020-12-19 00:28:53,222 ERROR ConfigHashSync - Failed to heartbeat to meta master to get configuration hash:
alluxio.exception.status.UnavailableException: Failed to connect to MetaMaster @ prod-xx-data-lake-002:19998 after 3 attempts
at alluxio.AbstractClient.connect(AbstractClient.java:274)
at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:385)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:349)
at alluxio.client.meta.RetryHandlingMetaMasterConfigClient.getConfigHash(RetryHandlingMetaMasterConfigClient.java:82)
at alluxio.client.file.ConfigHashSync.heartbeat(ConfigHashSync.java:82)
at alluxio.heartbeat.HeartbeatThread.run(HeartbeatThread.java:119)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: alluxio.exception.status.UnavailableException: Target Unavailable. GrpcChannelKey{ClientType=MetaMaster, ClientHostname=prod-xx-data-lake-002, ServerAddress=GrpcServerAddress{HostName=prod-xx-data-lake-002, SocketAddress=prod-xx-data-lake-002:19998}, ChannelId=99d6f5cf-15cc-4d25-b8a8-7de0a0a1ceb2}
at alluxio.grpc.GrpcChannelBuilder.build(GrpcChannelBuilder.java:226)
at alluxio.AbstractClient.connect(AbstractClient.java:227)
... xx more
Caused by: alluxio.exception.status.UnavailableException: io exception
at alluxio.exception.status.AlluxioStatusException.from(AlluxioStatusException.java:155)
at alluxio.exception.status.AlluxioStatusException.fromStatusRuntimeException(AlluxioStatusException.java:214)
at alluxio.exception.status.AlluxioStatusException.fromThrowable(AlluxioStatusException.java:199)
at alluxio.security.authentication.AuthenticatedChannelClientDriver.waitUntilChannelAuthenticated(AuthenticatedChannelClientDriver.java:187)
at alluxio.security.authentication.AuthenticatedChannelClientDriver.startAuthenticatedChannel(AuthenticatedChannelClientDriver.java:164)
at alluxio.security.authentication.ChannelAuthenticator.authenticate(ChannelAuthenticator.java:xx5)
at alluxio.grpc.GrpcChannelBuilder.build(GrpcChannelBuilder.java:212)
... 11 more
jiacheliu3
Alluxio Version: 2.2.0 Describe the bug Due to the expansion of the hadoop cluster and then performing hdfs data balancing, alluxio was down and could not be started To Reproduce Steps to reproduce the behavior (as minimally and precisely as possible)
Expected behavior A clear and concise description of what you expected to happen.
Urgency Describe the impact and urgency of the bug.
Additional context Add any other context about the problem here.