apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.01k stars 1.82k forks source link

[Bug] [Jdbc] An error is reported in version 2.3.5 for the types supported in version 2.3.4( db2/opengauss ) #6856

Open cooltnt opened 5 months ago

cooltnt commented 5 months ago

Search before asking

What happened

The "tinynt" in opengauss and the "LONGVARGRAPHIC" in db2 can be synchronized normally in 2.3.4, but when upgraded to 2.3.5, the prompt type is not supported

SeaTunnel Version

2.3.5

SeaTunnel Config

source table:
CREATE TABLE "check"."types" (
    "tinyint" int1
);

sink table:
CREATE TABLE public.types_opengauss (
    tinyint int2 NULL
);

--=============================================================

connector jar: postgresql-42.4.3.jar

env{
    "execution.parallelism" : 1,
    "job.mode" : "BATCH",
}
source{
    Jdbc {
        url = "jdbc:postgresql://******:5432/check"
        driver = "org.postgresql.Driver"
        user = "******"
        password = "******"

        query = """
            SELECT "tinyint"
FROM "check"."types"
        """
    }
}
transform{}
sink {
    Jdbc {
        "url" : "jdbc:postgresql://******:5432/test?stringtype=unspecified",
        "plugin_name" : "Jdbc",
        "database" : "test",
        "password" : "******",
        "driver" : "org.postgresql.Driver",
        "generate_sink_sql" : true,
        "user" : "******",
        "table" : "public.types_opengauss"
    }
}

--=========================================================================================================
source table:
CREATE TABLE "CHECK"."types" (
    LONGVARGRAPHIC LONG VARGRAPHIC(32700)
);

sink table:
CREATE TABLE public.types_db2 (
    "LONGVARGRAPHIC" varchar(32700) NULL
);

--=============================================================

connector jar: db2jcc-db2jcc4.jar

env{
        "execution.parallelism" : 1,
        "job.mode" : "BATCH",
}
source{
    Jdbc {
        url = "jdbc:db2://******:50000/CHECK"
        driver = "com.ibm.db2.jcc.DB2Driver"
        user = "******"
        password = "******"

        query = """
            SELECT "LONGVARGRAPHIC"
FROM "CHECK"."types"
        """
    }
}
transform{}
sink {
    Jdbc {
        "url" : "jdbc:postgresql://******:5432/test?stringtype=unspecified",
        "plugin_name" : "Jdbc",
        "database" : "test",
        "password" : "******",
        "driver" : "org.postgresql.Driver",
        "generate_sink_sql" : true,
        "user" : "******",
        "table" : "public.types_db2"
    }
}

Running Command

./bin/seatunnel.sh --config ./config/v2.opengauss.template -e local
./bin/seatunnel.sh --config ./config/v2.db2.template -e local

Error Exception

-- opengauss -----------------------------------------------------------------------------------------
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Jdbc'.
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:332)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
        ... 2 more
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-21], ErrorDescription:['Postgres' tables unsupported get catalog table,the corresponding field types in the following tables are not supported: '{"\n            SELECT \"tinyint\"\nFROM \"check\".\"types\"\n        ":{"tinyint":"int1"}}']
        at org.apache.seatunnel.common.exception.CommonError.getCatalogTablesWithUnsupportedType(CommonError.java:165)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.utils.JdbcCatalogUtils.getTables(JdbcCatalogUtils.java:111)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.<init>(JdbcSource.java:57)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceFactory.lambda$createSource$0(JdbcSourceFactory.java:78)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)
        ... 7 more

-- db2 -----------------------------------------------------------------------------------------
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'Jdbc'.
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:100)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:332)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:188)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:88)
        at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:156)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:149)
        ... 2 more
Caused by: org.apache.seatunnel.common.exception.SeaTunnelRuntimeException: ErrorCode:[COMMON-20], ErrorDescription:['UNKNOWN' table '
            SELECT LONGVARGRAPHIC
FROM "CHECK"."types"
        ' unsupported get catalog table with field data types '{"LONGVARGRAPHIC":"LONG VARGRAPHIC"}']
        at org.apache.seatunnel.common.exception.CommonError.getCatalogTableWithUnsupportedType(CommonError.java:151)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.utils.CatalogUtils.getCatalogTable(CatalogUtils.java:277)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.utils.CatalogUtils.getCatalogTable(CatalogUtils.java:243)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.utils.JdbcCatalogUtils.getCatalogTable(JdbcCatalogUtils.java:354)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.utils.JdbcCatalogUtils.getCatalogTable(JdbcCatalogUtils.java:347)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.utils.JdbcCatalogUtils.getTables(JdbcCatalogUtils.java:128)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource.<init>(JdbcSource.java:57)
        at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceFactory.lambda$createSource$0(JdbcSourceFactory.java:78)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:112)
        at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:73)

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.