search

WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
https://github.com/WeBankFinTech/DataSphereStudio-Doc
Apache License 2.0
3.09k stars 1k forks source link

sparksql脚本运行长时间等待，警告日志报“队列CPU资源不足，建议调小执行器个数。” 详细日志如下： #261

Open bobzhangfw opened 3 years ago

bobzhangfw commented 3 years ago

2020-11-21 16:37:29.036 [WARN ] [IDE_hadoopConsumerThread ] c.w.w.l.e.e.i.EntranceExecutorManagerImpl (55) [warn] - request engine failed! com.webank.wedatasphere.linkis.common.exception.WarnException: errCode: 30001 ,desc: 2020-11-21 16:37:29.037 WARN errCode: 111005 ,desc: When requesting queue resources(请求队列资源时)，The queue CPU resources are insufficient. It is recommended to reduce the number of actuators.(队列CPU资源不足，建议调小执行器个数。) ,ip: dataspherestudio ,port: 9104 ,serviceKind: ResourceManager ,ip: dataspherestudio ,port: 9105 ,serviceKind: sparkEngineManager at com.webank.wedatasphere.linkis.common.exception.ExceptionManager.generateException(ExceptionManager.java:50) ~[linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.transform.RPCConsumer$$anon$1.toObject(RPCConsumer.scala:63) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.BaseRPCSender$$anonfun$ask$1.apply(BaseRPCSender.scala:87) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorExchange.invoke(RPCInterceptorExchange.scala:32) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.CommonRPCInterceptor.intercept(CommonRPCInterceptor.scala:29) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.RetryableRPCInterceptor$$anonfun$intercept$1.apply(RetryableRPCInterceptor.scala:53) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) ~[linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.RetryHandler$class.retry(RetryHandler.scala:56) ~[linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.RetryableRPCInterceptor$RPCRetryHandler.retry(RetryableRPCInterceptor.scala:58) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.RetryableRPCInterceptor.intercept(RetryableRPCInterceptor.scala:53) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.CacheableRPCInterceptor.intercept(CacheableRPCInterceptor.scala:57) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.BroadcastRPCInterceptor.intercept(BroadcastRPCInterceptor.scala:74) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.BaseRPCSender.execute(BaseRPCSender.scala:79) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.BaseRPCSender.ask(BaseRPCSender.scala:83) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester.request(EngineRequester.scala:51) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager.createExecutor(EntranceExecutorManager.scala:70) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$askExecutor$1.apply(EntranceExecutorManager.scala:113) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$askExecutor$1.apply(EntranceExecutorManager.scala:112) ~[linkis-ujes-entrance-0.9.4.jar:?] at scala.Option.orElse(Option.scala:289) ~[scala-library-2.11.8.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager.askExecutor(EntranceExecutorManager.scala:112) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$4.apply(EntranceExecutorManager.scala:135) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$4.apply(EntranceExecutorManager.scala:135) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager.askExecutor(EntranceExecutorManager.scala:135) [linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcZ$sp$2.apply(FIFOUserConsumer.scala:113) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcZ$sp$2.apply(FIFOUserConsumer.scala:113) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3$$anonfun$apply$mcV$sp$1.apply$mcZ$sp(FIFOUserConsumer.scala:113) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.aux$1(Utils.scala:191) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.waitUntil(Utils.scala:204) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.waitUntil(Utils.scala:207) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3.apply$mcV$sp(FIFOUserConsumer.scala:112) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3.apply(FIFOUserConsumer.scala:108) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3.apply(FIFOUserConsumer.scala:108) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3.apply(FIFOUserConsumer.scala:133) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3.apply(FIFOUserConsumer.scala:107) [linkis-scheduler-0.9.4.jar:?] at scala.Option.foreach(Option.scala:257) [scala-library-2.11.8.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer.loop(FIFOUserConsumer.scala:107) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply$mcV$sp(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndError(Utils.scala:102) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer.run(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]

2020-11-21 16:37:32.068 [WARN ] [IDE_hadoopConsumerThread ] c.w.w.l.e.e.i.EntranceExecutorManagerImpl (55) [warn] - request engine failed! com.webank.wedatasphere.linkis.common.exception.WarnException: errCode: 30001 ,desc: 2020-11-21 16:37:32.037 WARN errCode: 111005 ,desc: When requesting queue resources(请求队列资源时)，The queue CPU resources are insufficient. It is recommended to reduce the number of actuators.(队列CPU资源不足，建议调小执行器个数。) ,ip: dataspherestudio ,port: 9104 ,serviceKind: ResourceManager ,ip: dataspherestudio ,port: 9105 ,serviceKind: sparkEngineManager at com.webank.wedatasphere.linkis.common.exception.ExceptionManager.generateException(ExceptionManager.java:50) ~[linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.transform.RPCConsumer$$anon$1.toObject(RPCConsumer.scala:63) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.BaseRPCSender$$anonfun$ask$1.apply(BaseRPCSender.scala:87) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorExchange.invoke(RPCInterceptorExchange.scala:32) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.CommonRPCInterceptor.intercept(CommonRPCInterceptor.scala:29) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.RetryableRPCInterceptor$$anonfun$intercept$1.apply(RetryableRPCInterceptor.scala:53) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) ~[linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.RetryHandler$class.retry(RetryHandler.scala:56) ~[linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.RetryableRPCInterceptor$RPCRetryHandler.retry(RetryableRPCInterceptor.scala:58) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.RetryableRPCInterceptor.intercept(RetryableRPCInterceptor.scala:53) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.CacheableRPCInterceptor.intercept(CacheableRPCInterceptor.scala:57) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.common.BroadcastRPCInterceptor.intercept(BroadcastRPCInterceptor.scala:74) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.interceptor.BaseRPCInterceptorChain.handle(RPCInterceptorChain.scala:35) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.BaseRPCSender.execute(BaseRPCSender.scala:79) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.rpc.BaseRPCSender.ask(BaseRPCSender.scala:83) ~[linkis-cloudRPC-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EngineRequester.request(EngineRequester.scala:51) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager.createExecutor(EntranceExecutorManager.scala:70) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$askExecutor$1.apply(EntranceExecutorManager.scala:113) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$askExecutor$1.apply(EntranceExecutorManager.scala:112) ~[linkis-ujes-entrance-0.9.4.jar:?] at scala.Option.orElse(Option.scala:289) ~[scala-library-2.11.8.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager.askExecutor(EntranceExecutorManager.scala:112) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$4.apply(EntranceExecutorManager.scala:135) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager$$anonfun$4.apply(EntranceExecutorManager.scala:135) ~[linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.entrance.execute.EntranceExecutorManager.askExecutor(EntranceExecutorManager.scala:135) [linkis-ujes-entrance-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcZ$sp$2.apply(FIFOUserConsumer.scala:113) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcZ$sp$2.apply(FIFOUserConsumer.scala:113) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3$$anonfun$apply$mcV$sp$1.apply$mcZ$sp(FIFOUserConsumer.scala:113) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.aux$1(Utils.scala:191) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.waitUntil(Utils.scala:204) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.waitUntil(Utils.scala:207) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3.apply$mcV$sp(FIFOUserConsumer.scala:112) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3.apply(FIFOUserConsumer.scala:108) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3$$anonfun$apply$3.apply(FIFOUserConsumer.scala:108) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3.apply(FIFOUserConsumer.scala:133) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$loop$3.apply(FIFOUserConsumer.scala:107) [linkis-scheduler-0.9.4.jar:?] at scala.Option.foreach(Option.scala:257) [scala-library-2.11.8.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer.loop(FIFOUserConsumer.scala:107) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply$mcV$sp(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer$$anonfun$run$1.apply(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryCatch(Utils.scala:48) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.common.utils.Utils$.tryAndError(Utils.scala:102) [linkis-common-0.9.4.jar:?] at com.webank.wedatasphere.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer.run(FIFOUserConsumer.scala:74) [linkis-scheduler-0.9.4.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_241] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]

wangpeng1462 commented 3 years ago

同问，我也遇到了这个问题

lucyilys commented 1 year ago

我也遇到了这个问题