make deal arbitration more scalable

[whyrusleeping]

Currently, deal arbitration is done per sector. This can get really expensive if clients store large amounts of data (large numbers of sectors). We should move towards giving clients the ability to either make deals that span multiple sectors (somehow, accumulators may help here) or at least efficiently arbitrate many deals at once.

[whyrusleeping]

The two major approaches I see for resolving this are broadly:

1. Reducing slashing complexity from O(Deal) to <O(Deals)
2. Reducing the number of deals a miner makes

Reducing slashing complexity to less than O(Deals)

Currently, if a miner were to drop all their data, it would require one message to the chain per deal the miner made (not even one per sector). This is pretty bad. One route of thinking here would be to find ways to slash a miner for multiple deals with a single invocation. Some initial ideas here may include accumulators, snarks, and other 'proving' systems. The most basic approach may just be to allow the ArbitrateDeal method to take multiple arguments. This doesnt gain us too much but its a start.

Reducing the number of deals a miner makes

If we reduce the number of deals that get made, then O(Deals) is less bad. One way we can do that is by allowing miners to make much larger deals. Currently deals are limited to the size of a single sector, if we allow larger sectors, then files that would have previously taken multiple deals to store could take just one.

Another approach to reducing the number of deals a storage miner makes, is to pack multiple deals into a single 'deal', to fill out an entire sector. Once we have repair miners, repair miners can perform this task for their clients, batching up smaller deals into bigger sectors for their storage miners.

[whyrusleeping]

cc @porcuquine @decentralion @nicola @bvohaska for discussion

[bvohaska]

At a very high level, it looks like we may need to choose one or more of the following options:

1. Somehow enable slashing to happen on actors off-chain (crazy idea)
2. Figure out how to aggregate slashes
3. Figure out how to compress slashes so they aren’t as much of a burden on-chain
4. Determine a method that forces miners to slash themselves (crazy idea)

I’m assuming the goal is to reduce the storage complexity of slashing on-chain with the bonus of reducing communication complexity. Ideally, we could do both. Is this accurate?

[whyrusleeping]

Roughly yes. re idea 1, slashing needs to happen on chain, in an ideal world it happens rarely, is cheap to do, and only needs to happen once per miner being slashed. 2 and 3 are discussed a little in my comment, 4 doesnt really help as things would still have to go on chain.

[whyrusleeping]

Concrete steps forward here for the near term:

• Allow a single signed message to invoke multiple methods (https://github.com/filecoin-project/go-filecoin/issues/320)
• Allow larger sectors so that deals for large amounts of storage span fewer sectors (not great, but it helps)

[mishmosh]

Closing as obsolete.