EricJoy2048 commented 2 years ago

SeaTunnel version: dev Hadoop version: Hadoop 2.10.2 Flink version: 1.12.7 Spark version: 2.4.3, scala version 2.11.12

Problems found to be fixed

Hive Sink Connector

The test source table is in https://github.com/apache/incubator-seatunnel/issues/2793

1. test text file format table

Because Hive Sink Connector not support array,map,struct datatype, so the test table will not contain array,map,struct.

create table test_hive_sink_text_simple(
     test_tinyint                          TINYINT,
     test_smallint                       SMALLINT,
     test_int                                INT,
     test_bigint                           BIGINT,
     test_boolean                       BOOLEAN,
     test_float                             FLOAT,
     test_double                         DOUBLE,
     test_string                           STRING,
     test_binary                          BINARY,
     test_timestamp                  TIMESTAMP,
     test_decimal                       DECIMAL(8,2),
     test_char                             CHAR(64),
     test_varchar                        VARCHAR(64),
     test_date                             DATE
     )
PARTITIONED BY (test_par1 STRING, test_par2 STRING);

1.1 test in flink engine

1.1.1 Job Config File

env {
  # You can set flink configuration here
  execution.parallelism = 3
  job.name="test_hive_source_to_hive"
}

source {
  Hive {
    table_name = "test_hive.test_hive_source_orc"
    metastore_uri = "thrift://ctyun7:9083"
    result_table_name = "tmp_table"
  }
}

transform {
   sql {
      sql = "select test_tinyint, test_smallint,test_int, test_bigint,test_boolean, test_float, test_double,test_string,test_binary,test_timestamp, test_decimal, test_char, test_varchar, test_date , 'p1' as test_par1, 'p2' as test_par2  from tmp_table"
   }
}

sink {
  # choose stdout output plugin to output data to console

  Hive {
    table_name = "test_hive.test_hive_sink_text_simple"
    metastore_uri = "thrift://ctyun7:9083"
    partition_by = ["test_par1", "test_par2"]
    sink_columns = ["test_tinyint", "test_smallint", "test_int", "test_bigint", "test_boolean", "test_float", "test_double", "test_string", "test_binary", "test_timestamp", "test_decimal", "test_char", "test_varchar", "test_date", "test_par1", "test_par2"]
  }
}

1.1.2 Submit job command

sh bin/start-seatunnel-flink-connector-v2.sh --config config/flink_hive_to_hive_simple.conf

TyrantLucifer commented 2 years ago

Please assign these bugs to me because of I'm the maintainer of Hive connector, thx.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.