search

datavane / tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
https://tis.pub
Apache License 2.0
1.04k stars 221 forks source link

创建mysql到Starrocks增量同步 #334

Closed ly-yangli closed 4 months ago

ly-yangli commented 4 months ago 
Compiling 1 source files to /opt/data/cfg_repo/streamscript/test/20240709132158/classes at 1720511456410
classpath:/opt/data/libs/plugins/tis-flink-dependency/WEB-INF/lib/*:/opt/data/libs/plugins/tis-flink-chunjun-starrocks-plugin/WEB-INF/lib/*:/opt/data/libs/plugins/tis-realtime-flink/WEB-INF/lib/*:/opt/data/libs/plugins/tis-realtime-flink/tis-scala-compiler-dependencies/*:/opt/data/libs/plugins/tis-flink-extends-plugin/WEB-INF/lib/*
cmd:  java -classpath /opt/data/libs/plugins/tis-flink-dependency/WEB-INF/lib/*:/opt/data/libs/plugins/tis-flink-chunjun-starrocks-plugin/WEB-INF/lib/*:/opt/data/libs/plugins/tis-realtime-flink/WEB-INF/lib/*:/opt/data/libs/plugins/tis-realtime-flink/tis-scala-compiler-dependencies/*:/opt/data/libs/plugins/tis-flink-extends-plugin/WEB-INF/lib/* scala.tools.nsc.Main -usejavacp -nobootcp -encoding utf8 -d /opt/data/cfg_repo/streamscript/test/20240709132158/classes /opt/data/cfg_repo/streamscript/test/20240709132158/src/main/scala/com/qlangtech/tis/realtime/transfer/test/TestListener.scala
prepare-compile in 0.0 s
compile in 5.3 s
pkgJar:/opt/data/libs/plugins/flink/test/WEB-INF/lib/test-incr.jar,tpi:/opt/data/libs/plugins/flink/test.tpi
✔✔ successful to publish Incr test Compile And Package'Incr test Compile And Package'
〇〇 start to publish Incr test Deploy'Incr test Deploy'
streamUberJar path:/opt/data/cfg_repo/streamscript/test/0/test-incr.jar
sinkFactories size:2
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_incr.xml
target xstream file is not exist:/opt/data/cfg_repo/tis_plugin_config/ap/test/com.qlangtech.tis.plugin.datax.SelectedTabExtend_batch.xml
start submit jar request,entryClass:com.qlangtech.plugins.incr.flink.TISFlinkCDCStart
create Flink app classloader:file:/opt/data/cfg_repo/streamscript/test/0/test-incr.jar,resolveOrder:CHILD_FIRST
start to explode archive:/opt/data/libs/plugins/flink/test.tpi
java.lang.RuntimeException: java.io.IOException: Unable to delete file: /opt/data/libs/plugins/test
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unable to delete file: /opt/data/libs/plugins/test
        at com.qlangtech.plugins.incr.flink.TISFlinkClassLoaderFactory.lambda$buildClientLoaderFactory$1(TISFlinkClassLoaderFactory.java:126)
        at org.apache.flink.client.ClientUtils.buildUserCodeClassLoader(ClientUtils.java:92)
        at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:145)
        at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:65)
        at org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
        at com.qlangtech.tis.plugins.flink.client.JarSubmitFlinkRequest.createFlinkJobProgram(JarSubmitFlinkRequest.java:105)
        at com.qlangtech.tis.plugins.flink.client.FlinkClient.submitJar(FlinkClient.java:80)
        at com.qlangtech.plugins.incr.flink.launch.clustertype.AbstractClusterType.deploy(AbstractClusterType.java:86)
        at com.qlangtech.plugins.incr.flink.launch.TISFlinkCDCStreamFactory.deploy(TISFlinkCDCStreamFactory.java:179)
        at com.qlangtech.plugins.incr.flink.launch.TISFlinkCDCStreamFactory.deploy(TISFlinkCDCStreamFactory.java:168)
        at com.qlangtech.tis.coredefine.module.action.TISK8sDelegate.deploy(TISK8sDelegate.java:141)
        at com.qlangtech.tis.coredefine.module.action.CoreAction.lambda$getFlinkJobWorkingOrchestrate$6(CoreAction.java:696)
        at com.qlangtech.tis.datax.job.JobResName$1.accept(JobResName.java:54)
        at com.qlangtech.tis.datax.job.SubJobResName.execute(SubJobResName.java:36)
        at com.qlangtech.tis.datax.job.JobResName.execSubJob(JobResName.java:71)
        at com.qlangtech.tis.coredefine.module.action.CoreAction.lambda$null$2(CoreAction.java:635)
        at com.qlangtech.tis.datax.job.ServerLaunchToken.writeLaunchToken(ServerLaunchToken.java:420)
        at com.qlangtech.tis.coredefine.module.action.CoreAction.lambda$launchIncrSyncChannel$3(CoreAction.java:630)
        at com.qlangtech.tis.datax.job.DefaultSSERunnable.execute(DefaultSSERunnable.java:112)
        at com.qlangtech.tis.coredefine.module.action.CoreAction.launchIncrSyncChannel(CoreAction.java:628)
        at com.qlangtech.tis.coredefine.module.action.CoreAction.doDeployIncrSyncChannal(CoreAction.java:603)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.qlangtech.tis.runtime.module.action.BasicModule.execute(BasicModule.java:176)
        at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:899)
        at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1544)
        at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
        at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethodWithDebugInfo(XWorkMethodAccessor.java:98)
        at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:90)
        at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1620)
        at ognl.ASTMethod.getValueBody(ASTMethod.java:91)
        at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
        at ognl.SimpleNode.getValue(SimpleNode.java:258)
        at ognl.Ognl.getValue(Ognl.java:470)
        at ognl.Ognl.getValue(Ognl.java:434)
        at com.opensymphony.xwork2.ognl.OgnlUtil$3.execute(OgnlUtil.java:401)
        at com.opensymphony.xwork2.ognl.OgnlUtil.compileAndExecuteMethod(OgnlUtil.java:453)
        at com.opensymphony.xwork2.ognl.OgnlUtil.callMethod(OgnlUtil.java:399)
        at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:438)
        at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:293)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:254)
        at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:250)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:179)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:263)
        at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:49)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.doIntercept(ConversionErrorInterceptor.java:142)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:137)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:137)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:201)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:67)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:133)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:89)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:243)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:101)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:142)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:160)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:175)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:121)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:167)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:203)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:196)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.qlangtech.tis.manage.common.valve.OperationLogInterceptor.doIntercept(OperationLogInterceptor.java:64)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.qlangtech.tis.manage.common.valve.TisExceptionInterceptor.doIntercept(TisExceptionInterceptor.java:89)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.qlangtech.tis.manage.spring.aop.AuthorityCheckAdvice.doIntercept(AuthorityCheckAdvice.java:109)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:137)
        at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
        at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
        at org.apache.struts2.factory.StrutsActionProxy.execute(StrutsActionProxy.java:48)
        at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:574)
        at org.apache.struts2.dispatcher.ExecuteOperations.executeAction(ExecuteOperations.java:79)
        at org.apache.struts2.dispatcher.servlet.StrutsServlet.service(StrutsServlet.java:80)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1411)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
        at com.qlangtech.tis.manage.common.DefaultFilter.doFilter(DefaultFilter.java:180)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:501)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.io.IOException: Unable to delete file: /opt/data/libs/plugins/test
        at com.qlangtech.tis.util.Util.deleteRecursive(Util.java:124)
        at com.qlangtech.tis.extension.impl.PluginManifest.explode(PluginManifest.java:437)
        at com.qlangtech.tis.extension.impl.PluginManifest.create(PluginManifest.java:179)
        at com.qlangtech.tis.extension.impl.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:96)
        at com.qlangtech.tis.extension.PluginManager.dynamicLoad(PluginManager.java:219)
        at com.qlangtech.plugins.incr.flink.TISFlinkClassLoaderFactory.lambda$buildClientLoaderFactory$1(TISFlinkClassLoaderFactory.java:115)
        ... 144 common frames omitted
Caused by: java.io.IOException: Unable to delete file: /opt/data/libs/plugins/test
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1338)
        at com.qlangtech.tis.util.Util.deleteRecursive(Util.java:122)
        ... 149 common frames omitted
Caused by: java.nio.file.FileSystemException: /opt/data/libs/plugins/test/WEB-INF/lib/.nfs00000000000444fc00000006: Device or resource busy
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
baisui1981 commented 4 months ago

看异常 您是在docker容器中部署增量实例吧，你看一下操作系统的卷是否无法正常删除？

Device or resource busy

Caused by: java.nio.file.FileSystemException: /opt/data/libs/plugins/test/WEB-INF/lib/.nfs00000000000444fc00000006: Device or resource busy
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)