juicedata / juicefs

JuiceFS is a distributed POSIX file system built on top of Redis and S3.
https://juicefs.com
Apache License 2.0
10.48k stars 919 forks source link

Support Storj trough native Libuplink library #1655

Open solracsf opened 2 years ago

solracsf commented 2 years ago

What would you like to be added:

https://www.storj.io/

Storj is an S3-compatible platform and suite of decentralized applications that allows to store data in a secure and decentralized manner. Your files are encrypted, broken into little pieces and stored in a global decentralized network of computers.

Go libraries are available: https://github.com/storj/uplink

Why is this needed:

Decentralized cloud storage represents a fundamental shift in the efficiency and economics of large-scale storage. Eliminating central control allows users to store and share data without reliance on a third-party storage provider. Decentralization mitigates the risk of data failures and outages while simultaneously increasing the security and privacy of object storage.

Please note that a S3-gateway is available, but this opts-in for a server-side encryption; using Uplink, end-to-end encryption is available and performance is superior.

davies commented 2 years ago

It looks like uplink is still in early phase, we would like to keep eye on it, don't depend on it for now.

solracsf commented 2 years ago

Hi, what do you call "early phase"? 🤔

amwolff commented 2 years ago

Of course, it depends on how one defines an early phase, but https://github.com/storj/uplink is a production-grade library at the moment. The S3-compatible object storage from Storj that JuiceFS can use as its backend uses uplink internally to communicate with the network.

solracsf commented 2 years ago

The main advantage of using Uplink library instead of S3 Gateway...is to avoid using the gateway :-) and introduce a SPOF.

seth-hunter commented 1 year ago

+1, very desirable feature. The ListObjects limitation of the MT gateway is problematic and the ST gateway workaround is resource intensive, slow, and seemingly buggy. Native uplink library interface seems promising.

eleaner commented 1 year ago

+1, fully agree

hannahkamundson commented 5 months ago

I would like to work on this!