New Revalidator API

[hannahhoward]

Goals

The current API for revalidation places a heavy burden on the markets / exchange layer tracking progress on a data transfer and inspecting every block sent for decisions about pausing.

We should make it easier for an exchange layer to only track the state relative to actual payments -- namely, how much has been paid, and given the current state of transfer, how much is owed.

Implementation

The new proposed API is as follows:

type CheckPoint struct{
    // First point after which to pause and ask for payment
    ByteOffset uint64
    // treat completing request as a checkpoint if it happens before next nextOffset
    PauseOnComplete bool
}

type Revalidator interface {

    // NextCheckPoint specifies the next point at which data transfer should
    // pause transfer
    // return values are:
    // has = are there remaining checkpoints for this request? If not continue to end
        // checkpoint = next checkpoint
    // err = abort request if not nil
    NextCheckPoint(
        channelID datatransfer.ChannelID,
                 bytesSoFar uint64
    ) (has bool, checkpoint Checkpoint, err error)

    // CheckpointReached allows an exchange to produce a request for payment
    // after a checkpoint has
    // been passed
    // Note: checkpoint is not exact for some protocols -- the byteOffs may be
    // slightly larger than
    // the one requested
    // return values are:
    // voucher result = information about payment requested, if any
    // err = error
    // - nil = pause request
    // - error = abort this request
    CheckpointReached(
        channelID datatransfer.ChannelID,
        actualByteOffset uint64,
    ) (datatransfer.VoucherResult, error)

    // EndOfDataReached allows for a final last request for payment
    // return values are:
    // voucher result = information about payment requested, if any
    // err = error
    // - nil = pause request
    // - error = terminate request with error
    EndOfDataReached(
        channelID datatransfer.ChannelID,
        actualByteOffset uint64,
    ) (datatransfer.VoucherResult, error)

    // Revalidate processes a new voucher (likely a payment) intended to resume
    // transfer
    // return values are:
    // voucher result = information about why the payment succeeded or failed
    // resume = should request resume? or stay paused for more revalidation
    // err = error
    // - nil = resume if resume = true
    // - error = abort this request
    Revalidate(
        channelID datatransfer.ChannelID,
        voucher datatransfer.Voucher,
        actualByteOffset uint64,
    ) (voucherResult datatransfer.VoucherResult, resume bool, err error)
}

This should allow the exchange layer to ignore all state tracking of bytes sent -- it only needs to track how much has been paid for a channel, and do calculations for amount owed based on price. This should allow us to remove all in memory tracking bytes sent / interval / etc in the markets layer

[hannahhoward]

Note that we also hope soon for data transfer channel IDs to become UUIDs

[hannahhoward]

related but not neccesarily dependent change -- https://github.com/ipfs/go-graphsync/issues/344

[willscott]

I would note that the disjoint between the planned pause / checkpoint apis proposed here, which are structured in terms of byte offsets in the stream is not an api that a client who's making a graphsync request in terms of a cid or selector is going to be able to interact with - they don't know how big the blocks / sub-dags are going to be exactly in many cases when making such a request.

[dirkmc]

Will I think for paid retrieval the client kind of has to know how big the thing they're downloading is, otherwise they don't know if they can afford to download it.

Hannah with regards to the proposed interface, I would favour an interface in which the layers don't have to query each other. This makes it easier to manage multiple threads, at the cost of each layer maintaining more state internally.

I'd also suggest a simplification in the protocol such that the provider doesn't ever ask for vouchers. Instead the client sends an updated voucher every time it receives data.

For example: The pricing scheme is a linear price per byte, eg 1 attoFIL / byte, with no up-front (eg unsealing cost). According to the pricing scheme the client and provider both know how much should be paid at each stage of the transfer, so there's no need for the provider to ask for vouchers. The client just sends another voucher each time it receives data. eg:

1. Client sends voucher for 1m attoFIL for first 1m bytes
2. Provider responds with bytes 0 - 500k.
3. Client sends voucher for 1m attoFIL.
4. Provider responds with bytes 500k - 1m.
5. Provider has reached 1m bytes so it pauses until it receives a voucher for more bytes from the client.
6. Client sends voucher for 2m attoFIL.
7. Provider responds with bytes 1m - 1.5m
8. Client sends voucher for 2m attoFIL.
9. Provider responds with bytes 1.5m - 2m
10. etc

If at any stage the client doesn't receive data for some timeout, the client resends the voucher for that tranche of data.

In terms of how this is implemented internally:

• Client
  • markets subscribes to notifications of incoming data on data-transfer layer
  • markets tells data-transfer layer to send voucher for 1m attoFIL
  • each time markets receives data, markets tells data-transfer layer to send 1m attoFIL
  • when received data reaches 1m bytes, markets tells data-transfer layer to send a voucher for 2m bytes
  • etc
• Provider
  • markets listens for received vouchers
  • when voucher for 1m bytes is received, tell data-transfer layer to keep sending until 1m bytes
  • when voucher for 2m bytes is received, tell data-transfer layer to keep sending until 2m bytes

So the interfaces on data-transfer are something like:

type DataReceiver interface {
    Subscribe(func(channelID, totalDataReceived))
    SendVoucher(channelID, voucher)
}

type DataSender interface {
    Subscribe(func(channelID, voucher))
    // SetDataLimit tells the DataSender to keep sending data until it reaches totalData bytes.
    SetDataLimit(channelID, totalData)
}

[willscott]

for paid retrieval the client kind of has to know how big the thing they're downloading is, otherwise they don't know if they can afford to download it.

If i was imagining what I would hope for, it would be something like saying 'pause after x blocks', with known constraints on the upper bound of block size.

[hannahhoward]

Will I think for paid retrieval the client kind of has to know how big the thing they're downloading is, otherwise they don't know if they can afford to download it.

There's a bigger meta issue here: we don't have good ways to estimate sizes for retrievals. Our whole protocol is based on a simple pricePerByte and right now I believe almost all of our size calculations are pretty made up based on the assumption of whole DAG. The only real way to calculate size for a partial retrieval is to run the selector.

I just want to raise this for future thinking. It's a meta question with selectors and IPLD. UnixFS has mechanisms for size calculations -- a "sum of sizes of the raw blocks underneath me" is built into the UnixFS data format.

[hannahhoward]

Hannah with regards to the proposed interface, I would favour an interface in which the layers don't have to query each other. This makes it easier to manage multiple threads, at the cost of each layer maintaining more state internally.

One thing I'm genuinely planning of here is that the payment layer lives out of process. This is something that's a genuine likelyhood when we move into IPFS. At the same time, I'm ok with Subscribes as long as they're non blocking -- I guess that pairs fine with websockets. I think your interface matches that.

I like your point that there's no need for a request for payment: you made an agreement, if you don't live up to it you should know why. I guess there's a trade off in being helpful as a retrieval provider ("here's why your transfer is stuck") and making the implementation complicated. The only other bummer is this is a change to the actual retrieval protocol. (the proposed interface I believe would work fine with the current protocol).

But all that said, your proposal looks super simple and perhaps worth it to implement.

I'd make a couple changes to enable to provider to be informative if they want to be:

What do you think?

type DataReceiver interface {
    Subscribe(func(channelID, lastVoucherResult, totalDataReceived))
    SendVoucher(channelID, voucher)
}

type DataSender interface {
    Subscribe(func(channelID, bestVoucher, totalDataSent))
    // SetDataLimit tells the DataSender to keep sending data until it reaches totalData bytes.
    SendVoucherResult(channelID, voucherResult)
    SetDataLimit(channelID, totalData)
}

[hannahhoward]

Also, I'm pretty sure the the subscribe calls effectively just reduce down to the current subscribe.

[hannahhoward]

The only thing that's needs is SendVoucherResult and SetDataLimit.

Also probably a good thing to consider how the first SetDataLimit gets called at the beginning of the request.

[dirkmc]

One thing I'm genuinely planning of here is that the payment layer lives out of process.

I'm in favour of this idea 👍 It should help enable different payment models (not necessarily voucher-based)

What's the purpose of SendVoucherResult(channelID, voucherResult) in the DataSender interface? Is it so that the data sender can ack that they've received the DataReceiver's voucher? When the DataSender sends data it's effectively an implicit ack of the DataSender's payment.

[hannahhoward]

It also allows the DataSender to give information about requesting payment -- the point here is to maintain protocol level compatibility if one wishes with the old interface -- which I'd like to do, since it's safe to assume both sides for retrieval may end-up at different versions. Arguably, that could become a retrieval protocol upgrade, but if I can avoid it it'd be nice to not worry about all that :)