apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.59k stars 1.67k forks source link

The database is the field where oracle partition_cloumn (id) is NUMBER; Size, scale 0, restapi submit-job error #6619

Open sandyustc opened 3 months ago

sandyustc commented 3 months ago

Search before asking

What happened

{ "status": "fail", "message": "ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - The current field is DecimalType containing decimals: 18 Unable to support" }

SeaTunnel Version

2.3.3

SeaTunnel Config

{
    "env": {
        "parallelism": 4,
        "job.mode": "batch",
        "checkpoint.interval": 10000
    },
    "source": [{
            "plugin_name": "jdbc",
            "url": "jdbc:oracle:thin:@xxx.xxx.11.181:1521:orcl",
            "driver": "oracle.jdbc.OracleDriver",
            "connection_check_timeout_sec": 100,
            "user": "xxx",
            "password": "xxx",
            "query": "select * from  table1",
            "partition_column": "id",
            "split.size": 10000

    }],
    "transform": [{
        "plugin_name": "Copy",
        "source_table_name": "xxxxx",
         "result_talbe_name": "xxxx2",
          "fields": {
               "id" : "xxxxx"
          }

    }

    ],
        "sink": [{
        "plugin_name": "Jdbc",
            "url": "jdbc:postgresql://xxx.xxx.11.xxx:5432/ptest",
            "driver": "org.postgresql.Driver",
            "user": "test",
            "password": "test",
            "generate_sink_sql": true,
            "database": "ptest",
            "table": "public.xxxx2",
            "schema_save_mode": "CREATE_SCHEMA_WHEN_NOT_EXIST",
            "data_save_mode": "APPEND_DATA",
            "primary_keys": ["id"]

    }]
}

Running Command

{
    "env": {
        "parallelism": 4,
        "job.mode": "batch",
        "checkpoint.interval": 10000
    },
    "source": [{
            "plugin_name": "jdbc",
            "url": "jdbc:oracle:thin:@xxx.xxx.11.181:1521:orcl",
            "driver": "oracle.jdbc.OracleDriver",
            "connection_check_timeout_sec": 100,
            "user": "xxx",
            "password": "xxx",
            "query": "select * from  table1",
            "partition_column": "id",
            "split.size": 10000

    }],
    "transform": [{
        "plugin_name": "Copy",
        "source_table_name": "xxxxx",
         "result_talbe_name": "xxxx2",
          "fields": {
               "id" : "xxxxx"
          }

    }

    ],
        "sink": [{
        "plugin_name": "Jdbc",
            "url": "jdbc:postgresql://xxx.xxx.11.xxx:5432/ptest",
            "driver": "org.postgresql.Driver",
            "user": "test",
            "password": "test",
            "generate_sink_sql": true,
            "database": "ptest",
            "table": "public.xxxx2",
            "schema_save_mode": "CREATE_SCHEMA_WHEN_NOT_EXIST",
            "data_save_mode": "APPEND_DATA",
            "primary_keys": ["id"]

    }]
}

Error Exception

{
    "status": "fail",
    "message": "ErrorCode:[API-01], ErrorDescription:[Configuration item validate failed] - PluginName: jdbc, PluginType: source, Message: org.apache.seatunnel.connectors.seatunnel.jdbc.exception.JdbcConnectorException: ErrorCode:[COMMON-06], ErrorDescription:[Illegal argument] - The current field is DecimalType containing decimals: 18 Unable to support"
}

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

java 1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

hawk9821 commented 3 months ago

What is the id field type for table1 ? is number(18,0) ?

sandyustc commented 3 months ago

What is the id field type for table1 ? is number(18,0) ? @hawk9821 thanks The ID field of the table is of type number(0,0)

hawk9821 commented 3 months ago

Problems with oracle data type conversion , A conversion logic error caused the oracle number type to be converted to Decimal(38,18). The expectation should be Long

sandyustc commented 3 months ago

Problems with oracle data type conversion , A conversion logic error caused the oracle number type to be converted to Decimal(38,18). The expectation should be Long @hawk9821 thanks

Oracle table field ID number type size and precision set to 0, should not be converted to decimal(38,18), why is it converted to this, what special requirements, if this is limited, that the column name for split data ,it can not be processed("partition_column": "id")

hawk9821 commented 3 months ago

image image The oracle number data type can store float data without specifying the precision, so seatunnel is treated as float and converted to Decimal(38,18). Change table field ID data type to number(x,0)

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.