search

iotaledger / iri

IOTA Reference Implementation
Other
1.15k stars 370 forks source link

rocksdb-cloud implementation #1762

Open taraszka opened 4 years ago

taraszka commented 4 years ago

Note Please check if there is a way to implement https://github.com/rockset/rocksdb-cloud as a new feature to the IRI

Description

RocksDB-Cloud is a C++ library that brings the power of RocksDB to AWS, Google Cloud and Microsoft Azure. It leverages the power of RocksDB to provide fast key-value access to data stored in Flash and RAM systems. It provides for data durability even in the face of machine failures by integrations with cloud services like AWS-S3 and Google Cloud Services. It allows a cost-effective way to utilize the rich hierarchy of storage services (based on RAM, NvMe, SSD, Disk Cold Storage, etc) that are offered by most cloud providers. RocksDB-Cloud is developed and maintained by the engineering team at Rockset Inc.

Motivation

This feature would let store IRI database on cloud service like AWS-S3, where the available storage is simply unlimited.

Requirements

A possibility to parametrize RocksDB database: local storage or cloud-based storage.

Am I planning to do it myself with a PR?

No.

karimodm commented 4 years ago

I think this is not viable due to the huge amount of IO that the DB can receive at certain stages of node startup or operations. The network and the connector will become the new bottleneck, which will be definitely more limiting than a dedicated IOPS SSD disk. Aside from the performance objection, I also find that relying on such a young project might be a serious mistake. @galrogo what do you think?