Albums screen: Should indicate summary of available downloads

[crspybits]

Currently, aside from Push Notifications, there's no way of knowing what albums have new info available in them without going into the specific album and refreshing. What would be better is to have a summary of new information available on the albums screen. E.g., with badges to indicate, say, the count of updates to files.

What then do we want for update info?

1) New media item count-- e.g., the number of new images. It's not really the number of files that are available to download that are needed, but rather the number of objects-- file groups.

2) Deletions-- how many objects need deleting in the album.

3) How many comment files need updating for existing media objects.

If the server supplied the number of non-deleted file groups and the number of deleted file groups, that would be most of this. But it would not get at the number of comments.

What if the server returned, per sharing group:

a) For non-deleted file groups: the last update Date for each file group, in a per file group format like: (file group UUID, last update date).

b) For deleted file groups, a count would be enough.

[crspybits]

I made a first effort on this, and it was informative but incomplete. I have learned that dates are insufficient. My first effort used the most recently changed server date across all file groups in a sharing group as a kind of serial number. That dealt with with quite a bit, but left an important use case missing.

When Neebla uploads a comment (or, more generally uploads a change to mutable file), it will subsequently download that update-- because when it updates, it doesn't know the updated version number. Currently, possibly multiple updates could be applied in the Uploader on the server when going to the next version number.

However, this leads to the issue that every time after uploading a new comment, you will have that album marked, erroneously in most cases, as needing a download. (Except for the rare case when someone else at a very similar time also commented on the same media item-- and both updates are included in the downloaded version change). Since adding a comment is a common case, and it will be unusual to have someone else comment at the same time, this needs to be fixed.

[crspybits]

I think the following additional changes are needed:

1) I'm going to have every single change applied by the Uploader increment the version number for a file. E.g., if there are M changes waiting for a file when the Uploader starts, and the file's current version number is N, then the resulting version number for that file will be N + M.

2)The index result when getting info in general for a sharing group (ie., an index request with a nil sharing group) will have more detailed information per file group. In specific, it will also contain the fileUUID+version number for each file contained in each sharing group. While it is tempting to only return fileUUID+version number for mutable files, that seems short sighted. New immutable files can be added to particular file groups at the clients initiative, and a client needs to be informed of such additions of immutable files to existing file groups.

3) On the client side:

a) We can know more accurately if files need downloading. If a file group returned from the server in the general index (without a sharing group given) is not known to the client, then of course, it needs to be downloaded. If the version number of any file, for a known file group, has been updated then that file group needs downloading.

b) We should be able to skip downloads for comment files having only changes caused by the local client. Since that will cause only a +1 increment to a comment file from its current version.

c) We ought to be able to provide much of this service from IOSBasics.

[crspybits]

One refinement: The information returned must be not just fileUUID+version, but fileUUID+version+userIdMakingLast change. This is needed to deal with race conditions where another client gets its change in earlier. For example:

time 0: Client 0 gets index

time 1: Client 1 uploads a change to file1, and this change is processed by the Uploader. Resulting in an incremented file version in the index.

time 2: Client 0 uploads a change to file1, and this change is delayed, not yet processed by the Uploader.

time 3: Client 0 gets index, and seeing that the version number of file1 has been incremented thinks the change was its change.

If at time 3, if client 0 could know the id of the user making the last change, it could have resolved the ambiguity.

[crspybits]

I think this above change to using fileUUID+version+userIdMakingLast only works for a single change, however.

If more than one change is made to a file, then further race condition complications come in. Since userIdMakingLast only allows a client to know the id of the user making the last change, if two prior changes were made race conditions can ensue.

time 0: Client 0 gets index

time 1: Client 1 uploads a change to file1, and this change is processed by the Uploader. Resulting in an incremented file version in the index.

time 2: Client 0 uploads a change to file1, and this change is processed by the Uploader. Resulting in an incremented file version in the index.

time 3: Client 0 uploads a change to file1, and this change is delayed, not yet processed by the Uploader.

time 4: Client 0 gets index, and seeing that the version number of file1 has been incremented by 2, and has its user id as the last change and thinks both of the recent changes were its changes.

[crspybits]

What if I had three cases:

1) The case of dates as I had at first.

2) Track on the client per file (or file group) if any change had been uploaded. Say, since the last download of that file (or file group). This would require resetting these states after downloads had been carried out per album.

3) Also return for an index request without a sharing group, per file group, a checksum for the contents of files. The server then would need to track this checksum per file. And the client would also need to track or be able compute this checksum.

(A) If the dates indicated no difference, then this would be sufficient evidence that the album had no downloadable change.

(B) If the dates indicated a change, and no change had been uploaded recently (since the last download of that file group), then this would be sufficient evidence that the album had a downloadable change.

(C) If the dates indicated a change, and a change had been uploaded recently by this client, the checksums would need to be evaluated.

If the underlying files on the server and the client were the same then the checksums would be the same-- so would indicate no download was needed.

If the underlying files on the server and the client were different then the checksums could be different or they could be the same. In the worst case, if the files on the client and server were different but the computed checksums were the same, we falsely wouldn't indicate to the client that a download was needed at the album level.

As Dany was talking about, if we we add a new file type that we use for purely book keeping purposes, and don't directly display to the user, then we'd have to add another complication. We might not want to indicate a download was available for the user if upon downloading that download showed nothing new to the user.

Another consideration: After a file upload for a mutable file completes fully, we could trigger a download of that file. That would take care of some of the uncertainty involved in case (C) above.

[crspybits]

This is what I have so far:

In iOSBasics-- when a vN upload is carried out a timer is started to periodically check for deferred uploads. This means that the deferredCompleted delegate method can get called without further user/client interaction. This has a side effect that the push notification method will get called for comments without further user/client interaction.

In Neebla, when the deferredCompleted delegate method is called, sync(sharingGroupUUID: album.sharingGroupUUID) is called to sync an album (it is possible that more than one sharing group/album is referenced in deferredCompleted but that's not dealt with yet).

In Neebla, when a sync completes for a specific album, a check is carried out to see if vN downloads are available. And if so, they are downloaded. This has the effect that when a user has added a comment, that comment will get downloaded. It also has the effect that it prioritizes the downloads of comments more generally.