apache / amoro

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
https://amoro.apache.org/
Apache License 2.0
836 stars 276 forks source link

[Improvement]: Support query on BaseStore #1304

Open baiyangtx opened 1 year ago

baiyangtx commented 1 year ago

Search before asking

What would you like to be improved?

According to the description of Arctic MixedFormat, a MixedFormat table consists of a Change Store and a BaseStore. Currently, both Spark engine and Trino engine support accessing the Change Store under MixedFormat, which is useful in data analysis scenarios. In order to further match the description of MixedFormat, we should also support accessing the BaseStore directly as a sub-table.

There are two scenarios for accessing the BaseStore directly:

  1. when better query performance is desired and high query responsiveness is required, but lower data freshness is acceptable, the BaseStore can be queried directly;
  2. when executing Timetravel, as MixedFormat does not yet support overall Timetravel, the BaseStore will be exposed as an Iceberg sub-table, allowing Timetravel operations to be performed directly on the BaseStore.

How should we improve?

Are you willing to submit PR?

Subtasks

Code of Conduct

majin1102 commented 1 year ago

Thanks for proposing this issue.

  1. Would you first clarify which search engines need to be supported?
  2. It will be nice if you explain why this function is needed
shidayang commented 1 year ago

Trino already supports querying the base table. like select * from {tableName}#base. reference:https://arctic.netease.com/ch/mpp/trino/#basestore

Is there any extra work to do?

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.