search

COOL-cohort / COOL

the source code of the COOL system
https://www.comp.nus.edu.sg/~dbsystem/cool/
Apache License 2.0
45 stars 16 forks source link

Add StorageService interface and a HDFS connector as an example #6

Closed hugy718 closed 2 years ago

hugy718 commented 2 years ago

This PR adds a storage service interface to COOL and the extension module hdfs-extensions as a sample implementation of it.

Adding the storage service implementation allows cool to offload the responsibility to keep data available and fault-tolerant to external storage services. This also facilitates the storage-compute separation and allows launching COOL workers on top of a shared storage layer in the future, which can be on cloud. More connectors to cloud services like AWS S3, and GCP Cloud Storage shall be supported later.

KimballCai commented 2 years ago

Please update the README to demonstrate how to run this code with an example.

hugy718 commented 2 years ago

There is an integration test example in the hdfs-extension folder. We shall later figure out how it can be incorporated into server. I will add some descriptions on README later.