Open steedmicro opened 6 months ago
IIRC when @allada and I talked about this a while ago we came to the conclusion that something like this might make sense:
blobstore
, i.e. a wrapper store where we can plug in different highly specialized/tuned backend implementations for AWS, GCP, Azure etc.One major hurdle I can currently see is that these stores can be tricky to test. Ideally we'll want some sort of short lived buckets for integration tests at some point. Technically I'd have a setup for CubeFS in an ad-hoc K8s cluster available, but while that might be useful for some API tests, that setup most likely can't mirror the actual behavior of AWS, GCP etc with respect to things like HTTP/HTTP2 and other not-so-obvious pitfalls.
@steed924 Consider what i mentioned here as a sort of 'wishful thinking out loud' rather than requirements - the current S3 store is not exactly in optimal shape, so anything in this area will probably be better than the current implementation 😅
Thanks. I will keep in mind that. +@aaronmondal.
FYI, while I was working on this issue, I encountered one problem while I was tryig to evaluate nativelink running with S3 bucket.
I was working on Ubuntu VPS and set aws credentials correctly and set environment variables for them also.
Even after I've tried run the command after bazel clean --expunge
the result was the same.
Running the nativelink by bazel run nativelink -- $(pwd)/nativelink-config/examples/s3_backend_with_local_fast_cas.json
seems to run server correctly.
But when I was going to evaluate the native link running by this command bazel test //... --remote_instance_name=main --remote_cache=grpc://127.0.0.1:50051 --remote_executor=grpc://127.0.0.1:50051 --remote_default_exec_properties=cpu_count=1
, it raises errors like this.
It raises an error while it tries to load the first chunk from ActionCache server, which is supported by S3 bucket.
I hope to hear your opinions. @aaronmondal. Thanks.
FYI, I've fixed the bug by myself.
The reason was because S3 bucket I was using was on us-east-1
region.
After I've updated the AWS_ENDPOINT_URL
and AWS_DEFAULT_REGION
environment varibles to us-east-1
, it seemed to be working fine.
Thanks.
We need to implement Google Cloud Bucket Store. We can refer Amazon S3 Bucket Store code -
nativelink-store/src/s3_store.rs
as a reference which implements similar logic we are going to implement. Unit tests and example config files need to be added.