WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
https://github.com/WeBankFinTech/DataSphereStudio-Doc
Apache License 2.0
3.09k stars 1k forks source link

azkaban会自己挂掉,参考以下日志,不排除也不确定是内存不够原因,改成-Xmx1024M再观察 #343

Open xiongjy2104 opened 3 years ago

xiongjy2104 commented 3 years ago

2021/07/10 15:35:51.454 +0000 ERROR [ExecutorManager] [Azkaban] QueueProcessorThread Interrupted. Probably to shut down. java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) at azkaban.executor.QueuedExecutions.fetchHead(QueuedExecutions.java:41) at azkaban.executor.ExecutorManager$QueueProcessorThread.processQueuedFlows(ExecutorManager.java:1241) at azkaban.executor.ExecutorManager$QueueProcessorThread.run(ExecutorManager.java:1221) 2021/07/10 15:35:51.484 +0000 WARN [AzkabanExecutorServer] [Azkaban] Shutting down executor... 2021/07/10 15:35:51.484 +0000 INFO [log] [Azkaban] Stopped SocketConnector@0.0.0.0:12321 2021/07/10 15:35:51.556 +0000 INFO [AzkabanWebServer] [Azkaban] kk thx bye. 2021/07/10 15:35:51.585 +0000 WARN [FlowRampManager] [Azkaban] Shutting down FlowRampManager now... 2021/07/10 15:35:51.586 +0000 WARN [FlowRunnerManager] [Azkaban] Shutting down FlowRunnerManager now... 2021/07/10 15:35:51.586 +0000 INFO [MBeanRegistrationManager] [Azkaban] Jmx MBean azkaban.jmx.JmxJettyServer:name=executorJetty unregistered. 2021/07/10 15:35:51.586 +0000 INFO [MBeanRegistrationManager] [Azkaban] Jmx MBean azkaban.execapp.jmx.JmxFlowRunnerManager:name=flowRunnerManager unregistered. 2021/07/10 15:35:51.586 +0000 INFO [MBeanRegistrationManager] [Azkaban] Jmx MBean azkaban.execapp.jmx.JmxFlowRampManager:name=flowRampManager unregistered. 2021/07/10 15:35:51.586 +0000 INFO [MBeanRegistrationManager] [Azkaban] Jmx MBean azkaban.execapp.jmx.JmxJobMBeanManager:name=jobJMXMBean unregistered. 2021/07/10 15:35:51.586 +0000 INFO [MBeanRegistrationManager] [Azkaban] Jmx MBean azkaban.execapp.jmx.JmxJobCallback:name=jobCallbackJMXMBean unregistered. 2021/07/10 15:35:51.586 +0000 WARN [FlowRunnerManager] [Azkaban] Deleting execution dir: /opt/dss/azkaban/executions Using Hadoop from /opt/hadoop-2.7.2 Using Hive from /opt/apache-hive-1.2.1-bin CLASSPATH: .:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64//lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64//jre/lib:bin/internal/../../lib/:bin/internal/../../extlib/:/opt/hadoop-2.7.2/conf:/opt/hadoop-2.7.2/:/opt/hadoop-2.7.2/share/hadoop/common/:/opt/hadoop-2.7.2/share/hadoop/yarn/:/opt/apache-hive-1.2.1-bin/conf:/opt/apache-hive-1.2.1-bin/lib/ java -Xmx512M -server -Djava.io.tmpdir=/tmp -Dexecutorport=12321 -Dserverpath=/opt/dss/azkaban -cp .:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64//lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64//jre/lib:bin/internal/../../lib/:bin/internal/../../extlib/:/opt/hadoop-2.7.2/conf:/opt/hadoop-2.7.2/:/opt/hadoop-2.7.2/share/hadoop/common/:/opt/hadoop-2.7.2/share/hadoop/yarn/:/opt/apache-hive-1.2.1-bin/conf:/opt/apache-hive-1.2.1-bin/lib/ azkaban.soloserver.AzkabanSingleServer -conf bin/internal/../../conf &

xiongjy2104 commented 3 years ago

在使用SQL和SCALA时候会挂掉 azkaban,但shell,python,jdbc都正常,难道这几个任务不走azkaban?