search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.47k stars 969 forks source link

[core][spark]support format table create, and implement format table for SparkCatalog #4223

Closed adrian-wang closed 1 month ago

adrian-wang commented 2 months ago

Purpose

Support format table creation, and add an implementation of format table for SparkCatalog.

Tests

Added SparkCatalogWithHiveTest

API and Format

No API changed.

Documentation

https://cwiki.apache.org/confluence/display/PAIMON/PIP-27%3A+Introduce+Format+Table+in+Paimon+Catalog

JingsongLi commented 2 months ago

See https://github.com/apache/paimon/pull/4243 , I think we can use this type to identify a format table.

melin commented 2 months ago

Support jdbc catalog?

adrian-wang commented 2 months ago

@melin need to extend jdbc table def before support it.

melin commented 2 months ago

@melin need to extend jdbc table def before support it.

After the jdbc catalog is supported, the hive metastore can be replaced.