search

datavane / tis

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
https://tis.pub
Apache License 2.0
1.04k stars 222 forks source link

[doris] 类型com.qlangtech.tis.plugin.datax.seq.SeqKey转型错误 #270

Closed baisui1981 closed 1 year ago

baisui1981 commented 1 year ago

由于扩展点 com.qlangtech.tis.plugin.datax.seq.SeqKey 是在 插件内部tis-datax-doris-plugin定义的 SeqKey.java

异常复现：

  1. 使用OffSeqKey 执行doris 全量导入
  2. 切换使用OnSeqKey执行全量导入 如下图 截屏2023-10-02 09 54 15

在assemble节点上出现如下错误：

Caused by: com.thoughtworks.xstream.converters.ConversionException: Cannot convert type com.qlangtech.tis.plugin.datax.seq.OnSeqKey to type com.qlangtech.tis.plugin.datax.seq.SeqKey
---- Debugging information ----
message             : Cannot convert type com.qlangtech.tis.plugin.datax.seq.OnSeqKey to type com.qlangtech.tis.plugin.datax.seq.SeqKey
class               : com.qlangtech.tis.plugin.datax.doris.DorisSelectedTab
required-type       : com.qlangtech.tis.plugin.datax.doris.DorisSelectedTab
converter-type      : com.qlangtech.tis.util.XStream2$TISCompositConvert
path                : /com.qlangtech.tis.plugin.KeyedPluginStore/plugins/com.qlangtech.tis.plugin.datax.doris.DorisSelectedTab/seqKey
line number         : 25
class[1]            : java.util.ArrayList
required-type[1]    : java.util.ArrayList
converter-type[1]   : com.thoughtworks.xstream.converters.collections.CollectionConverter
-------------------------------
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doUnmarshal(AbstractReflectionConverter.java:452)
    at com.qlangtech.tis.util.XStream2$TISCompositConvert.doUnmarshal(XStream2.java:252)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.unmarshal(AbstractReflectionConverter.java:277)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
    at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
    at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:117)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.addCurrentElementToCollection(CollectionConverter.java:99)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.populateCollection(CollectionConverter.java:92)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.populateCollection(CollectionConverter.java:86)
    at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
    at com.qlangtech.tis.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:402)
    at com.qlangtech.tis.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:344)
    at com.qlangtech.tis.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:278)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1421)
    at com.qlangtech.tis.util.XStream2.unmarshal(XStream2.java:70)
    at com.qlangtech.tis.extension.impl.XmlFile.unmarshal(XmlFile.java:119)
    at com.qlangtech.tis.plugin.PluginStore.load(PluginStore.java:399)
    ... 29 common frames omitted

故障定位

参考单元测试，TestDorisSelectedTab.java

由于执行了 TIS.clean()，导致 pluginManager中UberClassloader会重新初始化，导致之前已经加载的class会被重新加载，在内存中生成两个版本的class（虽然类名是相同的）

解决办法

对TIS.clean() 进行重载，引入参数 removeTIS：boolean，清除TIS过程中是否删除TIS实例。调用端可使用如下方法：

TIS.clean(false)