[Improve][Core][Exceptiom Management] Design Exception Management API

TyrantLucifer commented 1 year ago

This is a subtask of #2930.

The core API has two parts as the following shown:

SeaTunnelErrorCode

drawio

SeaTunnelErrorCode is a interface to define the error details for your module. This interface contains two important parts of information, part of the error code and part of the error description. When the user is using SeaTunnel and the exceptions thrown contain error codes, the error codes can be used to quickly locate the exceptions and reduce the cost of trial and error.

The following is an example of an implementation of this interface:

public enum CommonErrorCode implements SeaTunnelErrorCode {
    ENUM_ILLEGAL("Common-01", "This enum is not supported by SeaTunnel"),
    FILE_OPERATION_ERROR("Common-02", "File operation error"),
    JSON_OPERATION_ERROR("Common-03", "Json covert/parse operation error"),
    REFLECT_CLASS_OPERATION_ERROR("Common-04", "Reflect class operation error"),
    SERIALIZE_OPERATION_ERROR("Common-05", "Serialize class operation error");

    private final String code;
    private final String description;

    CommonErrorCode(String code, String description) {
        this.code = code;
        this.description = description;
    }

    @Override
    public String getCode() {
        return code;
    }

    @Override
    public String getDescription() {
        return description;
    }
}

The naming convention for error codes is as follows:

module name or connector name-01 module name or connector name-02

and so on.

This part will be the current public exception statistics to organize a public error code, for each sub-module more detailed error tips need to define their own.

SeaTunnelRuntimeException

SeaTunnelRuntimeException is an class that extended RuntimeException, there are three construction methods in it.

This is the SeaTunnel global unified base class exception. For different modules to define their own exception class and inherit from it.

When throw exceptions we need add SeaTunnelErrorCode in it.

Here are the details of the exceptions thrown by each module so far:

seatunnel-api

Num	Exception
1	NullPointerException
2	UnsupportedOperationException
3	OptionValidationException(user-defined)
5	FactoryException(user-defined)
8	IllegalArgumentException
0	TableNotExistException(user-defined)
0	DatabaseNotExistException(user-defined)
0	PrepareFailException(user-defined)
0	CatalogException(user-defined)

seatunnel-common

Num	Exception
1	ClosedException(user-defined)
1	ConfigRuntimeException(user-defined)
1	Exception
2	IllegalStateException
3	SerializationException(user-defined)
1	NoSuchMethodException
4	SeaTunnelException(user-defined)
4	IllegalArgumentException
15	RuntimeException

seatunnel-connectors-v2

Num	Exception
1	DatabaseNotExistException
1	EmptyXaTransactionException
1	GetElasticsearchVersionException
1	ParentNotDirectoryException
1	TableNotExistException
1	TransientXaException
1	UncheckedIOException
2	Exception
2	FileAlreadyExistsException
2	FilePluginException
3	HudiPluginException
4	BulkElasticsearchException
4	CatalogException
4	SQLException
6	FileNotFoundException
6	FTPException
12	IllegalStateException
32	IllegalArgumentException
34	IOException
48	PrepareFailException
56	UnsupportedOperationException
118	RuntimeException

seatunnel-core

Num	Exception
1	ParameterException
1	PluginClosedException
2	ConfigCheckException
2	ConfigRuntimeException
2	FileNotFoundException
2	TaskExecuteException
3	RuntimeException
5	CommandExecuteException
6	IllegalStateException
10	UnsupportedOperationException
13	IllegalArgumentException

seatunnel-engine

Num	Exception
1	AssertionError
1	CompletionException
1	RetryableHazelcastException
1	UnknownPhysicalPlanException
1	UnsupportedDeployTypeException
2	InvalidConfigurationException
2	UnknownActionException
2	UnknownFlowException
3	JobException
3	TaskRuntimeException
3	WrongTargetSlotException
4	SeaTunnelEngineException
7	JobDefineCheckException
10	IllegalArgumentException
10	UnsupportedOperationException
17	IllegalStateException
22	RuntimeException
24	CheckpointStorageException

seatunnel-formats

Num	Exception
1	IllegalArgumentException
1	IndexOutOfBoundsException
1	RuntimeException
2	IOException
3	JsonParseException
6	UnsupportedOperationException

seatunnel-plugin-discovery

Num	Exception
2	RuntimeException
2	UnsupportedOperationException

seatunnel-translation

Num	Exception
1	InvalidClassException
4	IllegalArgumentException
5	UnsupportedOperationException
33	RuntimeException

This shows that it is necessary to unify exception management.

Unified exception in API module is a simple implementation of this solution.

The final result is shown below:

ashulin commented 1 year ago

I think using enumerations is probably not a good idea, there could be some general exception classes.

TyrantLucifer commented 1 year ago

enumerations is used to tip the detail of error message, base on the newest discuss in community for every module to define a base exception class. I don't know if I understand correctly.

TyrantLucifer commented 1 year ago

@ashulin @Hisoka-X @EricJoy2048 @CalvinKirs @hailin0 I try to let everyone know. Thanks.

TyrantLucifer commented 1 year ago

According to the disscuss result that in Oct.25 meeting, SeaTunnel connector modules will use error code when throwing exception.

After the current code of statistics, summarized the following error code:

seatunnel-api

code	description
CONFIG_VALIDATION_FAILED	Configuration item validation failed
OPTION_VALIDATION_FAILED	Option item validation failed
CATALOG_INITIALIZE_FAILED	Catalog initialized failed
DATABASE_NOT_EXISTED	Database not existed
TABLE_NOT_EXISTED	Table not existed
FACTORY_INITIALIZE_FAILED	Factory initialization failed

seatunnel-common

code	description
ENUM_ILLEGAL	This enum is not supported by SeaTunnel
FILE_OPERATION_FAILED	File operation failed, such as (read,list,write,move,copy,sync) etc...
JSON_OPERATION_FAILED	Json covert/parse operation failed
REFLECT_CLASS_OPERATION_FAILED	Reflect class operation failed
SERIALIZE_OPERATION_FAILED	Serialize class operation failed
UNSUPPORTED_OPERATION	Unsupported operation
ILLEGAL_ARGUMENT	Illegal argument
UNSUPPORTED_DATA_TYPE	Unsupported data type
SQL_OPERATION_FAILED	Sql operation failed, such as (exceute,addBatch,close) etc...
TABLE_SCHEMA_GET_FAILED	Get table schema from clickhouse failed
FLUSH_DATA_FAILED	Flush batch data to sink connector failed
WRITER_OPERATION_FAILED	Sink writer operation failed, such as (open, close) etc...
READER_OPERATION_FAILED	Source reader operation failed, such as (open, close) etc...
HTTP_OPERATION_FAILED	Http operation failed, such as (open, close) etc...

connector-assert

AssertConnectorException

code	description
RULE_VALIDATION_FAILED	Rule validation failed

connector-clickhouse

ClickhouseConnectorException

code	description
TABLE_NOT_FOUND	Cannot get distributed table from clickhouse
TABLE_SCHEMA_GET_FAILED	Get table schema from clickhouse failed
CLUSTER_LIST_GET_FAILED	Get cluster list from clickhouse failed
FIELD_NOT_EXIST_IN_TABLE	Field not exist in table
TABLE_ENGINE_NOT_SUPPORTED	Table engine is not supported
SQL_OPERATION_FAILED	Sql operation failed, such as (exceute,addBatch,close) etc...
SHARD_KEY_NOT_FOUND	Shard key not found in table
PASSWORD_NOT_FOUND_IN_SHARD_NODE	Can’t find password of shard node
FLUSH_DATA_TO_FILE_FAILED	Flush data into clickhouse file failed
FILE_OPERATION_FAILED	File operation failed, such as (read,list,write,move,copy,sync) etc...
SSH_LOGIN_FAILED	Ssh login server failed
UNSUPPORTED_DATA_TYPE	Unsupported data type
JSON_OPERATION_FAILED	Json covert/parse operation failed

connector-amazondynamodb

AmazonDynamoDBConnectorException

code	description
UNSUPPORTED_DATA_TYPE	Unsupported data type
FLUSH_DATA_FAILED	Flush batch data to amazondynamodb failed

connector-elasticsearch

ElasticSearchConnectorException

code	description
BULK_RESPONSE_ERROR	Bulk response error
SCROLL_REQUEST_ERROR	Scroll request error
ILLEGAL_ARGUMENT	Illegal argument
UNSUPPORTED_DATA_TYPE	Unsupported data type
JSON_OPERATION_FAILED	Json covert/parse operation failed
GET_ES_VERSION_FAILED	Get elsticsearch version failed
GET_INDEX_DOCS_COUNT_FAILED	Get elsticsearch document index count failed

connector-email

EmailConnectorException

code	description
SEND_EMAIL_FAILED	Send email failed
FILE_OPERATION_FAILED	File operation failed, such as (read,list,write,move,copy,sync) etc...

connector-fake

FakeConnectorException

code	description
UNSUPPORTED_DATA_TYPE	Unsupported data type

connector-file

FileConnectorException

code	description
UNSUPPORTED_OPERATION	Unsupported operation
ILLEGAL_ARGUMENT	Illegal argument
FILE_OPERATION_FAILED	File operation failed, such as (read,list,write,move,copy,sync) etc...
UNSUPPORTED_DATA_TYPE	Unsupported data type
WRITER_OPERATION_FAILED	Orc or Parquet writer operation failed, such as (open, close) etc...

connector-hive

HiveConnectorException

code	description
ILLEGAL_ARGUMENT	Illegal argument
GET_HDFS_HOST_FAILED	Get hdfs host from hive metadata failed
GET_HIVE_META_FAILED	Get hive metadata failed

connector-http

HttpConnectorException

code	description
ILLEGAL_ARGUMENT	Illegal argument

connector-hudi

HudiConnectorException

code	description
ILLEGAL_ARGUMENT	Illegal argument
WRITER_OPERATION_FAILED	Parquet writer operation failed, such as (open, close) etc...
READ_PARQUET_META_FAILED	Read metadata from parquet file failed
KERBEROS_AUTHORIZED_FAILED	Kerberos authorized failed

connector-iceberg

IcebergConnectorException

code	description
UNSUPPORTED_DATA_TYPE	Unsupported data type
ILLEGAL_ARGUMENT	Illegal argument
SCAN_ICEBERG_SPLITS_FAILED	Failed to scan iceberg splits from a table
UNSUPPORTED_OPERATION	Unsupported operation
INVALID_RECORD_OFFSET	Invalid record offset

connector-influxdb

InfluxDBConnectorException

code	description
CONNECT_INFLUXDB_FAILED	Connect influxdb failed
UNSUPPORTED_DATA_TYPE	Unsupported data type
GET_COLUMN_INDEX_FAILED	Get column index of query result failed
ILLEGAL_ARGUMENT	Illegal argument
UNSUPPORTED_OPERATION	Unsupported operation

connector-iotdb

IotDBConnectorException

code	description
UNSUPPORTED_DATA_TYPE	Unsupported data type
ILLEGAL_ARGUMENT	Illegal argument
CONNECT_IOTDB_FAILED	Connect iotdb failed
CLOSE_IOTDB_FAILED	Close iotdb failed
FLUSH_DATA_FAILED	Flush batch data to iotdb failed
RETRY_TIME_EXCEED	The number of retries was exceeded
ILLEGAL_ARGUMENT	Illegal argument
UNSUPPORTED_OPERATION	Unsupported operation

connector-jdbc

JdbcConnectorException

code	description
CONNECT_DATABASE_FAILED	Connector database failed
GET_DATABASE_LIST_FAILED	Get database list failed
DATABASE_NOT_EXISTED	Database not existed
TABLE_NOT_EXISTED	Table not existed
UNSUPPORTED_DATA_TYPE	Unsupported data type
REFLECT_CLASS_OPERATION_FAILED	Reflect class operation failed
NO_SUITABLE_DRIVER	No suitable driver found
SQL_OPERATION_FAILED	Sql operation failed, such as (exceute,addBatch,close) etc...
XA_EXCEPTION	Xa operation failed, such as (commit, rollback) etc...

connector-kafka

KafkaConnectorException

code	description
REFLECT_CLASS_OPERATION_FAILED	Reflect class operation failed
CLOSE_KAFKA_SENDER_FAILED	Close kafka sender failed
ILLEGAL_ARGUMENT	Illegal argument
GENERATE_KAFKA_SPLIT_FAILED	Failed to generate kafka source split failed

connector-kudu

KuduConnectorException

code	description
ILLEGAL_ARGUMENT	Illegal argument
TABLE_SCHEMA_GET_FAILED	Get table schema from kudu failed
UNSUPPORTED_DATA_TYPE	Unsupported data type
GET_KUDU_SCAN_FAILED	Get the Kuduscan object for each split failed
CLOSE_KUDU_CLIENT_FAILED	Close kudu client failed
OPEN_KUDU_CLIENT_FAILED	Open kudu client failed

connector-mongo

MongoConectorException

code	description
UNSUPPORTED_DATA_TYPE	Unsupported data type
UNSUPPORTED_OPERATION	Unsupported operation

connector-neo4j

Neo4jConnectorException

code	desciption
UNSUPPORTED_DATA_TYPE	Unsupported data type

connector-pulsar

PulsarConnectorException

code	description
OPEN_PULSAR_ADMIN_FAILED	Open pulsar client failed
OPEN_PULSAR_CLIENT_FAILED	Open pulsar admin failed
PLUSAR_AUTHENTICATION_FAILED	Pulsar authentication failed
SUSCRIBE_TOPIC_FAILED	Subscribe topic from pulsar failed
GET_LAST_CURSOR_FAILED	Get last cursor of pulsar topic failed
GET_TOPIC_PARTITION_FAILED	Get partition information of pulsar topic failed
UNSUPPORTED_OPERATION	Unsupported operation
READER_OPERATION_FAILED	Source reader operation failed, such as (open, close) etc...

connector-redis

RedisConnectorException

code	description
UNSUPPORTED_DATA_TYPE	Unsupported data type
UNSUPPORTED_OPERATION	Unsupported operation
ILLEGAL_ARGUMENT	Illegal argument

connector-socket

SocketConnectorException

code	description
FLUSH_DATA_FAILED	Flush batch data to sink connector failed
OPEN_SOCKET_FAILED	Open socket client failed
CLOSE_SOCKET_FAILED	Close socket client failed

connector-starrocks

StarRocksConnectorException

code	description
FLUSH_DATA_FAILED	Flush batch data to sink connector failed
ILLEGAL_ARGUMENT	Illegal argument
UNSUPPORTED_DATA_TYPE	Unsupported data type

apache / seatunnel

[Improve][Core][Exceptiom Management] Design Exception Management API #3043