search

WeBankFinTech / DataSphereStudio

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
https://github.com/WeBankFinTech/DataSphereStudio-Doc
Apache License 2.0
3.04k stars 999 forks source link

[Bug] streamis执行flinksql报错：java.lang.IllegalStateException: Shutdown in progress #544

Open zhangxiaohui opened 2 years ago

zhangxiaohui commented 2 years ago

Search before asking

DSS Component

dss-plugins

What happened + What you expected to happen

在streamis中运行flinksql报错：java.lang.IllegalStateException: Shutdown in progress，同样的脚本可以在scriptis中正确执行。 2022-03-16 09:19:32.109 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 72 cancel - Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_161] at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_161] at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) [netflix-commons-util-0.1.1.jar:?] at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) [ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) [ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) [ribbon-loadbalancer-2.3.0.jar:2.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:280) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:214) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:92) [spring-cloud-context-2.2.1.RELEASE.jar:2.2.1.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:199) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170) [spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:949) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE]

2022-03-16 09:19:32.129 INFO [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 41 info - executorShutDownHook start to execute. 2022-03-16 09:19:32.129 WARN [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 50 warn - EngineConn not ready, do not shutdown 2022-03-16 09:19:32.131 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 67 cancel - Shutdown hook removed for: NFLoadBalancer-PingTimer-linkis-cg-linkismanager 2022-03-16 09:19:32.131 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 72 cancel - Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_161] at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_161] at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) [netflix-commons-util-0.1.1.jar:?] at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) [ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) [ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) [ribbon-loadbalancer-2.3.0.jar:2.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:280) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:214) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:92) [spring-cloud-context-2.2.1.RELEASE.jar:2.2.1.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:199) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) [spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170) [spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:949) [spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE]

2022-03-16 09:19:32.910 WARN [Linkis-Default-Scheduler-Thread-3] org.apache.flink.core.plugin.PluginConfig 69 getPluginsDir - The plugins directory [plugins] does not exist. 2022-03-16 09:19:34.490 ERROR [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor 58 error - Run code failed! org.apache.flink.table.api.TableException: Failed to execute sql at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:727) ~[flink-table-api-java-1.12.5.jar:1.12.5] at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:1107) ~[flink-table-api-java-1.12.5.jar:1.12.5] at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666) ~[flink-table-api-java-1.12.5.jar:1.12.5] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$2.get(FlinkCodeOnceExecutor.scala:112) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$2.get(FlinkCodeOnceExecutor.scala:111) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.client.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:164) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor.runCode(FlinkCodeOnceExecutor.scala:111) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(FlinkCodeOnceExecutor.scala:72) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$2.apply(FlinkCodeOnceExecutor.scala:72) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[scala-library-2.11.12.jar:?] at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) ~[scala-library-2.11.12.jar:?] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$1$$anonfun$run$1.apply$mcV$sp(FlinkCodeOnceExecutor.scala:72) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$1$$anonfun$run$1.apply(FlinkCodeOnceExecutor.scala:72) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$1$$anonfun$run$1.apply(FlinkCodeOnceExecutor.scala:72) ~[linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) [linkis-common-1.0.3.jar:1.0.3] at org.apache.linkis.engineconnplugin.flink.executor.FlinkCodeOnceExecutor$$anon$1.run(FlinkCodeOnceExecutor.scala:72) [linkis-engineconn-plugin-flink-1.0.3.jar:1.0.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster. at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:481) ~[flink-yarn_2.11-1.12.5.jar:1.12.5] at org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:81) ~[flink-clients_2.11-1.12.5.jar:1.12.5] at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1905) ~[flink-streaming-java_2.11-1.12.5.jar:1.12.5] at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135) ~[flink-clients_2.11-1.12.5.jar:1.12.5] at org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:55) ~[flink-table-planner-blink_2.11-1.12.5.jar:1.12.5] at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:711) ~[flink-table-api-java-1.12.5.jar:1.12.5] ... 22 more Caused by: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) ~[?:1.8.0_161] at java.lang.Runtime.addShutdownHook(Runtime.java:211) ~[?:1.8.0_161] at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1175) ~[flink-yarn_2.11-1.12.5.jar:1.12.5] at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:592) ~[flink-yarn_2.11-1.12.5.jar:1.12.5] at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:474) ~[flink-yarn_2.11-1.12.5.jar:1.12.5] at org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:81) ~[flink-clients_2.11-1.12.5.jar:1.12.5] at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1905) ~[flink-streaming-java_2.11-1.12.5.jar:1.12.5] at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135) ~[flink-clients_2.11-1.12.5.jar:1.12.5] at org.apache.flink.table.planner.delegation.ExecutorBase.executeAsync(ExecutorBase.java:55) ~[flink-table-planner-blink_2.11-1.12.5.jar:1.12.5] at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:711) ~[flink-table-api-java-1.12.5.jar:1.12.5] ... 22 more stdout.log

Relevent platform

DSS1.1+Linkis1.0.3+Streamis0.1.0

Reproduction script

oracl-cdc

Anything else

No response

Are you willing to submit a PR?

wushengyeyouya commented 2 years ago

Please submit this issue to Streamis.