Major Enhancement: Add local write support to Store (includes Offline support)

[eyalgu]

Currently Store only supports reading data, it does not provide any support for writing local updates back up to the server.

We would like to add write support to Store that considers offline updates as a first class citizen requirement.

In order to achieve the above we will need to deal to introduce support for things like:

• pending operations
• conflict resolution
• updating the remote
• etc.

Some discussion has been started on this subject at https://paper.dropbox.com/doc/Offline-First-Thoughts-Doc--Ap0nQdYi5xHxXTFl3K~WSGmoAg-JCrUclQot6HBhydtFYxNY

[eyalgu]

Note that lifecycle scoping gets tricker with uploads and we will need to make some decision here.

As long as we're only supporting reading data, it makes sense to have the store operate in a well defined scope like an Activity.

Once we add updates into the mix we'll need to add support for WorkManager or an equivalent to run the uploads even if the user navigates away from the read scope.

[digitalbuddha]

Maybe I'm misunderstanding, why wouldn't suspend fun Store.update(key) be sufficient since a user must launch that function inside of a scope. That way we leave it up to the consumer to either launch in a global scope or something like a lifecycle scope. Are you thinking something closer to suspend fun Store.update(key, scope) ?

[yigit]

i actually think most people will have the store as a global thing, like a singleton. (because repositories are usually singletons). That is also necessary to use cache across activities.

Should we do this in a branch though? I feel like it is a big undertaking to get into Store 4.0

[eyalgu]

@digitalbuddha - Here's what I'm thinking. For offline support to be a truly first class citizen, we would want to drain the LocalModification queue whenever network became available That operation would outlive the scope of whatever the user will call update on. suspend fun Store.update(key) will probably return whenever the local modification was recorded rather then when it was updated on the server.

Does that make sense?

[digitalbuddha]

It does! Could we maybe leverage the same scope that share uses? It is global by default unless user overrides

[eyalgu]

I think the scope used for share (reading) is quite different than the flow used for writing for 2 main reasons: 1) While the default for the read flow is GlobalScope it has a very clear API for changing this. In fact it seems as though the point of this API is to tie Store to a more specific scope like an activity. 2) Write's "scope" is not actually a coroutine scope. We probably want the app to launch in the background in order to upload updates when network is available.

[eyalgu]

going to work on branch store-update-support

[eyalgu]

^ This branch should now require PRs to commit like master so we can just merge it at the end

[eyalgu]

I've created a milestone for this issue so I can start breaking it down into smaller issues. I'll file them all under https://github.com/dropbox/Store/milestone/1

[bisakhmondal]

Hi everyone, Bisakh this side. I am interested in working on this feature as part of GSoC22. A while ago, I went through the dropbox paper link you have shared here and on the gsoc idea page. I have some thoughts based on the doc content. It would be awesome if you could take a look when you are free.

[Pasting my comment from the dropbox docs]

​​Hi everyone, Bisakh this side. I’d love to work on this issue as a part of GSoC 2022. ​​I have some doubts. |||>> ​​objects that can be re-run​ when items are fetched from the server ​​ ​​So IIUC, the modifications must be commutative, right? ​​ ​​|||>> developer calls the API to re-apply local changes on the object ​​ ​​Does it mean that we need to define the modification as Idempotent operations or we are going to keep some metadata (lamport clock ? ) to choose which modification to apply and which one not to (if it’s already been applied)? ​​ ​​Any thoughts on using CRDT (Conflict Free Replicated Data Type) based algorithms. For example, if we can represent the underlying data source as a JSON document, simple algorithms like RGA might be a good start for optimistic conflict resolution.

Thank you so much! Please share your thoughts : )

[bisakhmondal]

ping @digitalbuddha