reflector.go should implement a server-side cache to reduce S3 traffic

[kauffj]

• [x] buy ovh server and set it up with ansible
• [x] write code to cache blobs on disk between S3 download and serving to client
• [x] blobs should be split into subdirectories so we don't have too many files in one dir
• [x] configure relatime so we can efficiently track which blobs were accessed least recently
• [x] write code to clear cache to keep it under X (45TB?) in size
• [x] file issue in SDK to separate reflector upload and download servers
• [ ] look into benefits/drawbacks of using another filesystem than ext4
• [x] set up monitoring for this server

[chron0]

I would also really love to see it being able to work independently of S3, so that all storage would be on the server running the reflector instead of having to rely on AWS (at least as an option).

[lyoshenka]

@chron0 this issue talks about s3 because we happen to use s3. however you can use any data store. look at the store package. the BlobStore interface is used throughout the rest of the code, so you can swap in any store you want. we even have a disk store already written.

[chron0]

Hmmm, gotta look deeper into it. I like the idea of running a "public reflector/storage node", especially if there is also some kind of reward system to do so. In the meantime my focus is on not relying on my local client to keep my content online. Tom already gave me the franklin package to hack around a while back but it's php and this is go and I'd love to have like my "personal" reflector storing my content so it will be available via p2p, even if my client is offline. Would you say this is already within the scope of the code so that I can dig deeper into it?

[lyoshenka]

Our vision with reflector was to have a federated network of hosts that store data for users and get paid for providing that data when users request it. This is the start of that work, and it sounds like that's what you're looking for.

A lot of the pieces are still missing. The DHT doesn't work quite right, and there's no federation or incentive to host files. I'm glad to hear that you want to play with this even though its not finished. Some of this work is on our roadmap for 2020 but we haven't gotten to it yet. If you want to get your hands dirty, I'm happy to hash out concrete next steps with you sooner.

[chron0]

IC. I dont mind getting my hands dirty, but I'm not gods greatest gift as a programmer. As for the "vision", ye, I've read that between the lines or was at least kinda hoping for something of a decentralized nature to have headless storage in the network. We can try this "you give me concrete small bits" to solve and I give it a shot.

I think both use cases would be great to solve in this:

• Have federated general nodes with a store/credit system as an incentive for people to run
• Have headless non-local storage for creators to keep their content online without lbry-desktop open