search

danthegoodman1 / icedb

An in-process Parquet merge engine for better data warehousing in S3
https://blog.danthegoodman.com/icedb-v3--third-times-the-charm
Other
128 stars 5 forks source link

On-disk support (network fs) #117

Open danthegoodman1 opened 1 year ago

danthegoodman1 commented 1 year ago

Have a mode where it operates off disk instead of S3 in case of using something like AWS fsx lustre. That might provide better managed performance, especially for reading the log.

loicalleyne commented 7 months ago

Is this for both the metastore and the parquet data files? I suppose this would work with a local RAID drive setup as well?

danthegoodman1 commented 7 months ago

Yeah it would be really for any locally mounted disk most likely, and yes both would be supported as they are currently colocated under one parent directory for the table

danthegoodman1 commented 4 months ago

Also useful for JuiceFS

danthegoodman1 commented 4 months ago

Should be able to chose location for log and data separately, so could chose to have log on disk but data in s3