search

Shannon-Data / ShannonBase

A MySQL HTAP Database, Open Source version of MySQL Heatwave, Powered by AI.
https://www.shannonbase.org
Other
16 stars 6 forks source link

feat(shannon): Reading and writing open table format #31

Open ShannonBase opened 1 year ago

ShannonBase commented 1 year ago

To support reading and writing the open table format, such as parquets. which will enable Shannonbase lakehouse processing abilities. Nowdays, volume of data is getting larger, and the type of data is going to be more versatile, such as structured data, semi-structured data, unstructured data.

MySQL Heatwave, Databricks, etc. are all support data lakehouse.

RingsC commented 11 months ago

Using AFSL(Abstract File System Layer) or specific engine to operate on specific engine?

for example: create table test (xxx) engine = innodb ENGINE_ATTRIBUTE='{"file": [{"region":"", "namespace":"", "bucket":"", "name":""}]}';

ShannonBase commented 5 months ago

using create tablespace. When we ar creating a table space, we can specify the path of space file path. then using this tablespace to create a table.

by this, we can locate shannon rapid table to a NVME disk.

ref: https://dev.mysql.com/doc/refman/8.0/en/create-tablespace.html