search

apache / gravitino-playground

A playground to experience Gravitino
Apache License 2.0
31 stars 31 forks source link

[#34] add Gravitino Spark connector to playground #45

Closed FANNG1 closed 2 months ago

FANNG1 commented 5 months ago

add spark connector to playground

fixes: #34

FANNG1 commented 5 months ago

@qqqttt123 , could you help to review when you free?

jerqi commented 5 months ago

There are some issues.

  1. Trino and Spark will init the metalake twice.
  2. Is there the conflict if we want to use Spark rest catalog and Spark connector at the same time?
  3. Do you need a case about Spark connector?
FANNG1 commented 5 months ago

There are some issues.

  1. Trino and Spark will init the metalake twice.

the script checks the exist of metalake and catalogs before creating it.

  1. Is there the conflict if we want to use Spark rest catalog and Spark connector at the same time?

no, I changed the rest catalog name to avoid conflict.

  1. Do you need a case about Spark connector?

no, it's out of the scope of this PR

jerqi commented 5 months ago

You should include the case. Because it's the verification about the Spark connector.

FANNG1 commented 5 months ago

You should include the case. Because it's the verification about the Spark connector.

this pr only focus on add spark connector to playground, we could use another pr to do it

jerqi commented 5 months ago

You should include the case. Because it's the verification about the Spark connector.

this pr only focus on add spark connector to playground, we could use another pr to do it

You should add the document or you add the steps of your test at least. Otherwise, I don't how to verify this pull request.

FANNG1 commented 5 months ago

You should include the case. Because it's the verification about the Spark connector.

this pr only focus on add spark connector to playground, we could use another pr to do it

You should add the document or you add the steps of your test at least. Otherwise, I don't how to verify this pull request.

so, I propose another PR in Gravitino playground document?

FANNG1 commented 3 months ago

@jerqi , update the document, please help to review again

jerqi commented 3 months ago

Will it influence the demo of Iceberg REST catalog? Is it clear for user to use Spark connector and REST service catalog. It is a confusing for user.

jerqi commented 3 months ago

You should submit the document to the main repo, too.

FANNG1 commented 3 months ago

Will it influence the demo of Iceberg REST catalog? Is it clear for user to use Spark connector and REST service catalog. It is a confusing for user.

The original document makes Spark and Trino use same catalog name iceberg_catalog, but they are different, maybe we could add some comment to tell user iceberg_catalog and rest_catalog refer to the same underlying catalog, WDYT?

FANNG1 commented 2 months ago

@jerqi , I have tested spark trino and Iceberg REST service part, please help to review again