Delaying deletions: Stale file versions

[crspybits]

Currently, file deletions are carried out relatively quickly. However, the following scenario can occur:

1) Client A gets a FileIndex. 2) Client B does a file deletion or an upload change, which ends up generating a new file version. 3) Those client B changes get carried by the server Uploader. 4) Client A now tries to do a file download of one of the files changed/deleted by Client B. They will fail because of the Client B changes. Client A at best will need to do another FileIndex, get updated file version(s), and re-attempt the download.

This could also go on perpetually-- in which Client A effectively gets starved because of continuous updates to the files it's trying to access.

What would be better is if we don't do deletions of older file versions immediately. We should acknowledge that it's better for a client to get a somewhat stale version of a file rather than possibly being starved of getting any file updates. What's important is that a client get a consistent set of updates. (e.g., all files in a file group get changed together)

This will require changing the ServerUploader code so that it doesn't carry out deletions immediately. Rather they are postponed. I think the strategy to use is to postpone them for a period of time. This period of time needs to be the maximum likely amount of time it takes a Client A to (a) get a FileIndex from the server, and (b) do a series of downloads to, say, get all files in a file group. An hour might be reasonable. More than 24 hours seems too much. The longer this interval, the more storage can be taken up in an owning users cloud storage purely for these stale files. For files that get updated frequently, this could be appreciable.

We'll likely need to keep a record on the server of the stale versions of files that (a) need to be deleted later, and (b) that can still be downloaded by a client with a stale FileIndex. Our database FileIndex tracks the actual most recent file versions. We may need a new table that tracks stale file versions for this purpose.

On a file download, we'll need to both check the FileIndex to see if the client is downloading that file, and this stale file table to see if they are downloading a stale file.

I'm not sure what is going to actually carry out the deletions of stale file versions that have aged beyond usefulness. Perhaps the ServerUploader can do this as part of its operation.

[crspybits]

I'm seeing now why I chose to keep file versions in the DownloadFileRequest-- it's so that the client can have control over the specific version of a file they select. It does imply the need for this deferred deletion mechanism here.

[crspybits]

It seems like two changes are needed: 1) Implementing this delayed file deletion mechanism 2) Implement a retry mechanism on the client which includes removing the DownloadFileTracker (see iOSBasics). This is needed in the extreme case when the file version being downloaded needs updating on the basis of the current FileIndex obtained from the server.