search

WeBankFinTech / Exchangis

Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources
https://github.com/WeBankFinTech/Exchangis.git
Apache License 2.0
448 stars 208 forks source link

[Bug] exchangis执行数据交换成功,但结束任务时报错 #392

Closed coombe closed 1 year ago

coombe commented 1 year ago

Search before asking

Exchangis Component

exchangis-plugins

What happened + What you expected to happen

1

sqoop 引擎报如下错误:

2023-02-10 10:23:34.252 INFO [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 85 executorShutDownHook - executorShutDownHook start to execute. 2023-02-10 10:23:34.252 WARN [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 91 executorShutDownHook - had stop, do not shutdown 2023-02-10 10:23:34.262 INFO [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 85 executorShutDownHook - executorShutDownHook start to execute. 2023-02-10 10:23:34.262 WARN [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 91 executorShutDownHook - had stop, do not shutdown 2023-02-10 10:23:34.263 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 67 cancel - Shutdown hook removed for: NFLoadBalancer-PingTimer-linkis-cg-engineconnmanager 2023-02-10 10:23:34.263 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 72 cancel - Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_311] at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_311] at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) ~[netflix-commons-util-0.3.0.jar:0.3.0] at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_311] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_311] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_311] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311] at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:286) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:220) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:93) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:205) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:949) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE]

2023-02-10 10:23:34.271 INFO [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 85 executorShutDownHook - executorShutDownHook start to execute. 2023-02-10 10:23:34.271 WARN [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 91 executorShutDownHook - had stop, do not shutdown 2023-02-10 10:23:34.272 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 67 cancel - Shutdown hook removed for: NFLoadBalancer-PingTimer-linkis-cg-linkismanager 2023-02-10 10:23:34.272 INFO [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 72 cancel - Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_311] at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_311] at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) ~[netflix-commons-util-0.3.0.jar:0.3.0] at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0] at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_311] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_311] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_311] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_311] at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:286) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:220) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:93) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE] at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:205) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:949) ~[spring-context-5.2.22.RELEASE.jar:5.2.22.RELEASE]

2023-02-10 10:23:37.277 WARN [DiscoveryClient-0] com.netflix.discovery.TimedSupervisorTask 90 run - task supervisor shutting down, can't accept the task

Relevent platform

exchangis-1.1.2-webank/dss-1.1.1/linkis-1.3.1/sqoop-1.4.6/centos7.9

Reproduction script

无

Anything else

No response

Are you willing to submit a PR?

seawindh commented 1 year ago

看看字段映射里头是不是有字段类型不匹配了，可以只保留一个string字段试试看

coombe commented 1 year ago

看看字段映射里头是不是有字段类型不匹配了，可以只保留一个string字段试试看

感谢您。应该不是这个问题，我看了一下，我这4个字段，都是字符串类型的，字段名称完全相同。而且，数据已经传输到目标表里了。我看日志是关闭任务时失败了，感觉应该是linkis升级之后，exchangis的某些地方需要适配。

Li-GL commented 1 year ago

同样的问题，exchangis dev-1.1.1, linkis 1.3.1

XXwhite commented 1 year ago

同样的问题，数据同步成功，返回状态失败，在sqoop引擎日志的最后有两个报错

jefftlin commented 1 year ago

It had been resolved by pr #408 . The latest version is 1.1.2, the latest dss version is 1.1.2, and the latest linkis version is 1.4.0