Closed leozc closed 4 months ago
I created this draft PR: https://github.com/coinbase/chainstorage/pull/43
If it's on the correct direction, I'll continue working on it.
For local implementation we are using AWS's localstack.
@jiezhang, localstack works for now - but here are my concerns
@leozc For local run using GCP storage backends, looks like we may use their emulators? https://cloud.google.com/sdk/gcloud/reference/emulators
https://cloud.google.com/sdk/gcloud/reference/beta/emulators
The emulators do not include BigTable
, unless we want to use firestore
.
And I think the local implementation @leozc mentioned was not only for local development, but also a solution to baremetal deployment or using private cloud without similar products.
It's available in the beta emulators: https://cloud.google.com/sdk/gcloud/reference/beta/emulators
Don't get me wrong. I'm NOT against this idea of building a storage abstraction. But we should do it step by step. Initially we should focus on building the GCP abstraction and leveraging its emulators for local runs and integration tests.
It's available in the beta emulators: https://cloud.google.com/sdk/gcloud/reference/beta/emulators
This is great, I was finding an alternative to BigTable for integration tests, turns out they are already supporting it.
@bestmike007 we can consider this as done?
Yes, I think so.
This is a large issue that may need to be broken down further:
Currently, ChainStorage is bound to AWS dependency
Why? In order to help ChainStorage be more portable, we need to break down these hard-wired dependencies. A possible solution is to provide abstract interfaces for these storage solutions and implement the adaptor to different cloud providers accordingly.
We also explored a driver-level compatibility layer (e.g., CHainStorage continue to use S3 library interfaces, and we adopt the S3 driver to different cloud provider solution) - but it is a no-go due to complexity.
Tickets so far:
PS We should have a local implementation - Blob Storage -> File, SQL (lite?) -> KV, and a simple table for DLQ, for testing and possibly local production use cases.