[Bug] [Doris 2 Doris] Unsupported doris type: JSON

oldwolf1984 commented 3 weeks ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

编写脚本，读取Doris源库中一个数据表，数据表中一个名叫data的字段，类型是json类型，期望读取data.属性1，写入Doris另一个目标表中，目标表是一个普通字符串类型字段微信图片_20240610143806 微信图片_20240610143932

SeaTunnel Version

2.3.5

SeaTunnel Config

无

Running Command

执行脚本

Error Exception

2024-06-10 14:19:37,125 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Doris'.
    at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:332)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)
    at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
    at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
    ... 2 more
Caused by: org.apache.seatunnel.api.table.catalog.exception.CatalogException: ErrorCode:[API-03], ErrorDescription:[Catalog initialize failed] - Unsupported doris type: JSON
    at org.apache.seatunnel.connectors.doris.util.DorisCatalogUtil.fromDorisType(DorisCatalogUtil.java:260)
    at org.apache.seatunnel.connectors.doris.catalog.DorisCatalog.getTable(DorisCatalog.java:233)
    at org.apache.seatunnel.connectors.doris.source.DorisSourceFactory.createSource(DorisSourceFactory.java:86)
    at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:111)
    at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
    ... 7 more

2024-06-10 14:19:37,125 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - 
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
    at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
    at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Doris'.
    at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:332)
    at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)
    at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
    at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
    at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
    ... 2 more
Caused by: org.apache.seatunnel.api.table.catalog.exception.CatalogException: ErrorCode:[API-03], ErrorDescription:[Catalog initialize failed] - Unsupported doris type: JSON
    at org.apache.seatunnel.connectors.doris.util.DorisCatalogUtil.fromDorisType(DorisCatalogUtil.java:260)
    at org.apache.seatunnel.connectors.doris.catalog.DorisCatalog.getTable(DorisCatalog.java:233)
    at org.apache.seatunnel.connectors.doris.source.DorisSourceFactory.createSource(DorisSourceFactory.java:86)
    at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:111)
    at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Carl-Zhou-CN commented 3 weeks ago

@oldwolf1984 dev has enhancements to doris's type that you can try

EricJoy2048 commented 3 weeks ago

Hi, In seatunnel 2.3.5, the doris source connector support read json column. And you can get data.XBNO use SQL Transform.

The job config can like this:

source {
  Doris{
    xxx
    doris.read.field = "id, data"

  }
}

transform{
  Sql {
    query = "select id, id as oid, data.XBNO as materialNo from book_cms_dwt"
  }
}

Sink {
  xxx
}

apache / seatunnel