StarRocks / starrocks-connector-for-apache-spark

Apache License 2.0
36 stars 53 forks source link

[Feature] Support to load array type #74

Closed banmoy closed 1 year ago

banmoy commented 1 year ago

What type of PR is this:

Which issues of this PR fixes :

Fixes #

Problem Summary(Required) :

Support to load array data type to starrocks. Here is an example

StarRocks DDL

CREATE TABLE `array_tbl` (
  `id` INT NOT NULL,
  `a0` ARRAY<STRING>,
  `a1` ARRAY<ARRAY<INT>>
) ENGINE=OLAP
PRIMARY KEY(`id`)
DISTRIBUTED BY HASH(`id`)
PROPERTIES (
  "replication_num" = "1"
);

Spark DataFrame

val data = Seq(
   |  (1, Seq("hello", "starrocks"), Seq(Seq(1, 2), Seq(3, 4))),
   |  (2, Seq("hello", "spark"), Seq(Seq(5, 6, 7), Seq(8, 9, 10)))
   | )
val df = data.toDF("id", "a0", "a1")
df.write
     .format("starrocks")
     .option("starrocks.fe.http.url", "127.0.0.1:8038")
     .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9038")
     .option("starrocks.table.identifier", "test.array_tbl")
     .option("starrocks.user", "root")
     .option("starrocks.password", "")
     .option("starrocks.column.types", "a0 ARRAY<STRING>,a1 ARRAY<ARRAY<INT>>")
     .mode("append")
     .save()

Note that

Checklist: