Optimize sync and make it real-time

[frankhinek]

Background/Context

During a sync pull() operation:

• a MessagesGet request is sent over the network to the remote DWN for every messageCid that needs to be sync'd
• if the message returned by MessagesGet does NOT contain encodedData , then an additional RecordsRead must be sent over the network to fetch the data from the datastore, if any

During a sync push() operation:

• a MessagesGet request is sent to the local DWN for every messageCid that needs to be sync'd
• if the message returned by MessagesGet does NOT contain encodedData , then an additional RecordsRead must be processed by the local DWN to fetch the data from the datastore, if any

An obvious optimization we can and will pursue is to batch the first step such that all messageCids are included in a single MessagesGet request sent either to the local DWN or over the network to a remote DWN. This is a change to an application or SDK using dwn-sdk-js, so as such, will be implemented first in the web5-js SDK.

However, as currently implemented, there isn't a way to optimize the second step.

Question to Research and Propose Alternative Solutions

How might we modify MessagesGet (or add a new method) such that fewer over the network requests are needed to retrieve the message metadata and data payload for each message that is queued for sync?

[csuwildcat]

Can we just use/require HTTP/2 framed multiplexing for multiple file fetch?