Allow farmers to heal files on the network with missing shards - reconstruction with erasure

MeijeSibbel commented 7 years ago

Package Versions

Replace the values below using the output from npm list storj. Use npm list -g storj if installed globally.

storj-bridge@0.6.5 /home/gordon/Code/storj-bridge
└── storj@0.6.5  -> /home/gordon/Code/storj-core

Replace the values below using the output from node --version.

v4.2.3

Expected Behavior

Please describe the program's expected behavior. Include an example of your usage code in the back ticks below if applicable.

Currently there are plenty of files on the network with missing shards because the farmer went offline before replication could be triggered or even because uploads are blocked, also preventing replication. The result of this is files stored on the Storj network which are weak and much more vulnerable to permanent loss of the entire file. Reconstructing a missing shard with erasure is a computer intensive task, specially if hundreds or thousands of files have to be healed in a short time span. Although the bridge could do this, when the network scales healing of files by the bridge will pose a computational nightmare, specially if thousands of files have to be healed in a very short time span. Allowing farmers to voluntarily donate their system resources (computational + bandwidth) to act as file healers to reconstruct the missing shards is a much better and easier to scale solution than healing on the bridge. As monetary incentive the healer would either get a micro-payment bonus for healing the file or the farmer is allowed to himself store the reconstructed missing shard. Other parameters such as speed of reconstruction and speed of transfer of the reconstructed shard can be taken to benchmark the healer nodes.

Actual Behavior

Please describe the program's actual behavior. Please include any stack traces or log output in the back ticks below.

When a farmer goes offline right after receiving a shard from the renter the file will always remain incomplete than the risk of permanent loss is (x) times greater.

Steps to Reproduce

Please include the steps the reproduce the issue, numbered below. Include as much detail as possible.

...
...
...

tempestb commented 7 years ago

I'm pretty sure parity reconstruct happens in the client tool. All the shards are downloaded and then the file is reconstructed. So... I suppose you could write something that reconstructs the file with Storj Share and then uploads that missing shard back to the network, maybe, but I'm not sure it is necessary to do so. Since you'd have to lose 2/3rds of all shards right now, which is pretty difficult to do.

braydonf commented 7 years ago

Right now the plan is to have the bridge perform this task. We specifically made it so that parity is after encryption to enable shard recovery without having access to private keys.

braydonf commented 7 years ago

MeijeSibbel commented 7 years ago

Won't bridge reconstructions lead to problems with scalability?, what if you have millions of shard in the future that have to be reconstructed?, you would need insane amounts of computing power and not even to mention bandwidth.

littleskunk commented 7 years ago

That's why we will still create mirrors because they are easy to recreate.

The problem with mirror creation is simple. Let's say out of 10000 farmer 6 are going offline at the same time. The chance for a lost shard is close to 0 but if you multiply it with 1M shards you can be sure at least one shard will be lost. That's why we have to reconstruct these few lost shards.

Am 22.08.2017 1:59 nachm. schrieb "Meije Sibbel" notifications@github.com:

Won't bridge reconstructions lead to problems with scalability?, what if you have millions of shard in the future that have to be reconstructed?, you would need insane amounts of computing power and not even to mention bandwidth.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Storj/core/issues/712#issuecomment-324004720, or mute the thread https://github.com/notifications/unsubscribe-auth/AGCjzvxCTo8MpoP8n0sDiUiKKhkKAFIAks5sasKrgaJpZM4Ov7tW .

braydonf commented 6 years ago

It may be possible for us to provide the means for farmers to coordinate and repair missing shards, if the information about missing shards is made available to them, and then they can claim a "bounty" on the contract for that shard, since the original contract was abandoned. Could be an interesting area to explore and help with scalability issues with shard recovery.

RichardLitt commented 6 years ago

👋 Hey! Thanks for this contribution. Apologies for the delay in responding!

We've decided to rearchitect Storj, so that we can scale better. You can read more about this decision here. This means that we are entirely focused on v3 at the moment, in the storj/storj repository. Our white paper for v3 is coming very, very soon - follow along on the blog and in our Rocketchat.

As this repository is part of the v2 network, we're no longer maintaining this repository. I am going to close this for now. If you have any questions, I encourage you to jump on Rocketchat and ask them there. Thanks!

storj-archived / core