File Sync

[sanderpick]

As per team call regarding the plan for camera roll sync:

• Added files are always pinned locally, regardless of the schema directive
• Clients (mobile, CLI, etc.) can then "release" a file once it's data has been remotely pinned to a cafe
• This action can be recorded on the file index (as a non-shared column), perhaps a list of cafe peer ids that currently pin the file
• If the schema allows, the local IPFS node can actually unpin the file data
• When listing files / feed, we can include this pin info which can be shown in a nice way to the user (num. pin icons, etc.)

[sanderpick]

In an effort to open up some of this design, I want to share my current thinking here. What we are after is something like GraphSync.

Goals

• Pin a graph and the block pointing to it before broadcasting this new state to thread members
• A cafe should recursively pin the top-level target hash of a DAG so as to simplify record keeping and unpinning.
• A cafe should be able to track the amount of data is being pinned by each of its clients

The current implementation is based on the idea of a "cafe request", and specifically "store" type cafe requests. Other types are for snapshot-ing and un-snapshot-ing threads, delivering a message to a peer, and un-storing. For each node / object in a graph, a store request is created. Currently, there is no intelligent grouping of these requests. (Actually, we do currently have a concept of grouping but it needs to change slightly as proposed below).

Proposal

A file(s) input creates a graph that needs to be synced. To avoid unbounded request sizes from giant file groups, we can break up the graph as follows:

• A "Sync Group" contains all the requests needed to fully sync the graph and the block to a single cafe
• The collection of nodes that do not have direct data links can be handled in one multipart request
• Each meta/content pair can be handled in one multipart request
• The resultant block can be handled in one request

This implies that the pinning of the block and the publishing of the block to the thread must be split into distinct steps. Once the Sync Group has been completely synced, the block can be "released" to the thread. This results in the following (this is already implemented):

1. A "block message" is created for each member and added to the "block outbox" (a queue)
2. An attempt is made to send the message (encrypted block) directly to the recipient peer
3. If this fails, it is removed from this queue and used to create a cafe inbox request and is added to the cafe outbox (it does not need to be grouped with others)

Note: In practice, the block that is actually released may be newer since more recent actions may have generated blocks that were more quickly synced (perhaps not part of a large group).

Desktop

On desktop, we should be able to eventually use GraphSync when it's ready. Our current implementation functions over libp2p and performs work similar to bitswap's want list where blocks are only sent if they are needed. This is determined by a pre-request round-trip where the cafe can remove hashes from the the list of blocks that will be sent.

Mobile

On mobile, all requests must be handled by the background upload services on iOS/Android. This makes doing something like the want list approach a good bit harder. So, the first pass here does not care about bandwidth duplication. This means that we can make use of multipart requests for each "request group" above.

[sanderpick]

Call notes:

• Assume that retries are handled by the native layers on mobile
• Need to expose a "failed" endpoint that will delete the sync group and block
• Sync Group is keyed by the block id
• Avatars on profiles should return the whole block
• Notifications can be sent once req. groups are marked at complete containing the total sync group stats
• A notification can be sent when the block is "released"
• Stats can be queried at any time by sync group id
• A UI that is essentially a list of the sync groups and their statuses was discussed

Call participants: @asutula, @sanderpick

[sanderpick]

Currently working on implementing a slightly tweaked interface for the cafe reqs. This can support desktop and mobile request handlers.

type CafeRequestStore interface {
    Queryable
    Add(req *pb.CafeRequest) error
    Get(id string) *pb.CafeRequest
    List(offset string, limit int) *pb.CafeRequestList
    ListGroups(offset string, limit int) []string
    ListCompletedSyncGroups() []string
    SyncGroupStatus(syncGroupId string) *pb.CafeRequestGroupStatus
    UpdateStatus(id string, status pb.CafeRequest_Status) error
    UpdateGroupStatus(group string, status pb.CafeRequest_Status) error
    Delete(id string) error
    DeleteByGroup(groupId string) error
    DeleteBySyncGroup(syncGroupId string) error
    DeleteByCafe(cafeId string) error
}