search

filecoin-project / go-data-transfer

Data Transfer Shared Component for go-filecoin & go-lotus
Other
39 stars 17 forks source link

Data transfer resumption for large data transfers exceeds message size limits #235

Open aarshkshah1992 opened 3 years ago

aarshkshah1992 commented 3 years ago

To support transfer resumption, a data-transfer requestor sends a Graphsync DoNotSendCids message to the responder with ALL the CIDs received so far. This solution does NOT scale as sending ALL cids recieved for large data transfers causes the message size to exceed the message size limits set by the Graphsync protocol.

We need to redesign how we resume large data transfers.

aarshkshah1992 commented 3 years ago

cc @whyrusleeping who has been running into this with Estuary resuming 32G data transfers.