[Paper] Local-first software: You Own Your Data, in spite of the Cloud

[imnotteixeira]

https://www.scopus.com/record/display.uri?eid=2-s2.0-85076749929&origin=resultslist&sort=plf-f&src=s&st1=real+time+communication+web+offline&st2=&sid=2a870bc34ff2970b9cae781a860c8735&sot=b&sdt=b&sl=50&s=TITLE-ABS-KEY%28real+time+communication+web+offline%29&relpos=3&citeCnt=4&searchTerm=

https://www.inkandswitch.com/media/local-first/local-first.pdf

Cloud apps like Google Docs and Trello are popular because they enable real-time collaboration with colleagues, and they make it easy for us to access our work from all of our devices. However, by centralizing data storage on servers, cloud apps also take away ownership and agency from users. If a service shuts down, the software stops functioning, and data created with that software is lost. In this article we propose local-first software, a set of principles for software that enables both collaboration and ownership for users. Local-first ideals include the ability to work offline and collaborate across multiple devices, while also improving the security, privacy, long-term preservation, and user control of data. We survey existing approaches to data storage and sharing, ranging from email attachments to web apps to Firebase-backed mobile apps, and we examine the trade-offs of each. We look at Conflict-free Replicated Data Types (CRDTs): data structures that are multi-user from the ground up while also being fundamentally local and private. CRDTs have the potential to be a foundational technology for realizing local-first software. We share some of our findings from developing local-first software prototypes at the Ink & Switch research lab over the course of several years. These experiments test the viability of CRDTs in practice, and explore the user interface challenges for this new data model. Lastly, we suggest some next steps for moving towards local-first software: for researchers, for app developers, and a startup opportunity for entrepreneurs. © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.

[imnotteixeira]

Proposes ideals for local-first software, local-first software being the idea that the data on users' local machines is the primary when compared to servers, no the other way around, as it often happens in web apps.

1. No spinners(loading) - data is changed locally, and sync occurs quietly in the background
   • a pattern is mentioned which is just a workaround - Optimistic UI - to show the changes immediately, while sending them to the server, which would need to be reverted in case of error, for example
2. Multi device support (work everywhere) - section 3 discusses sync technologies for this
3. Optional network: Since local-first applications store the primary copy of their data in each device’s local filesystem, the user can read and write this data anytime, even while offline. It is then synchronized with other devices sometime later, when a network connection is available
4. Seamless Collaboration- section 4 discusses tech for real time collab in local-first architecture
5. data stays forever- the data is independent of the company/provider, since it is stored locally
6. secure and private by default - by having the data on each user's device locally, database tampering is not as harmful, as you'll have lots of backups. Moreover, by using e2e encryption, the server can only save encrypted data, thus making it private, as it can only be decrypted by the users who are allowed to do so.
7. ownership - the user owns the data, the server/product cannot block the access in any form since the data is local

[imnotteixeira]

Besides CRDTs, the more established technology for real-time collaboration is Operational Transformation [122] (OT), as implemented e.g. in ShareDB [118]

[imnotteixeira]

Provides https://github.com/automerge/automerge and https://github.com/automerge/hypermerge as JS tools for using CRDTs

[imnotteixeira]

Compares multiple applications in the way they handle the local-first ideals

• Files + email attachments
• Google Docs
• Trello
• Pinterest
• Dropbox
• Git + Github as well as technologies to implement them
• Thin client (web app)
• Thick client (mobile app)
• Backend as a service (e.g. firebase)
• CouchDB (multi master db that allows each node to mutate the DB and sync the changes with other nodes - meant to run on servers)
  • PouchDB tries to achieve this running on the end user devices

Also mentions CRDTs as a foundational technology to achieve the local-first ideals. CRDT stands for Conflict-Free Replicated Data Types, which are general-purpose data structures similar to the common lists and maps but built for multi-user environments from the beginning.

CRDT merges changes from multiple users when possible, however it does not handle changes to the same element in the structure, in that case it keeps track of the conflicts for the application to deal with them, allowing for custom conflict resolution techniques.

CRDTs are generic enough that they can sync over any communication connection like server, peer-to-peer networks, Bluetooth, and others.

Automerge is a JS CRDT implementation

Furthermore, the authors built some prototypes using Electron, JavaScript and React using CRDTs to verify that using CRDTs was already an option in order to build local-first software for the web and desktop applications.

The main conclusions were that CRDT worked reliably, while integrating easily with the other tools and seamlessly merging data. Also, they verified that the user experience was splendid, as it allowed for offline work and sync when possible, giving the feeling of "data ownership" to the user. Finally, they state that this technology combines well with Functional Reactive Programming (FRP), a paradigm which renders the view based on a function that receives data. If the data changes, it "reacts" and redraws. A popular framework using this model is React, but there are others such as Vue.js or Flutter. With such a tool, by synching the data with a CRDT and reacting to the changes in terms of UI, a real-time experience is easily achieved.

It added that CRDTs might have a performance problem if used to same many changes (since they essentially save all the history). That is something that must be taken into account when using that technology and designing the system.