Request of helpfull hints : Automerge Sync Protocol Implementation

[raphael10-collab]

In the attempt to re-write https://github.com/cudr/slate-collaborative/blob/master/packages/backend/src/AutomergeBackend.ts to make it fit with the latest automerge version changes, I'm trying to implement what has been outlined here: https://github.com/automerge/automerge/blob/main/SYNC.md

This is what I've done so far (forgive me for eventual errors):

interface peer_docId_hash {
  [docId: string]: string | Automerge.SyncState
}

interface syncStatesType {
  [peerId: string]: peer_docId_hash // a hash of [source][docId] containing in-memory sync state
}

type docsHandledByPeer = string[]

interface peerType {
  peerId: string,
  docs?: docsHandledByPeer,
}

class AutomergeBackend {
  peers: peerType[] // peers discovered by hyperswarm listening to the same "subject"

  syncStates: syncStatesType = {}

  backends = {} // a hash by [docId] of current backend values : ??????????

  syncPeersListFeed = (peer: peerType): syncStatesType => {
    let sst: syncStatesType = {}

    let peersKeysList = Object.keys(this.peers[0])

    for (let j = 0; j < this.peers.length; j++) {
      let peer_j= this.peers[j]
      let peer_j_0 = peer_j[peersKeysList[0]]
      sst[peer_j_0] = {}
    }
    return sst

  constructor(peers) {
    this.peers = peers
    this.syncStates = this.syncPeersListFeed(this.peers[0])
  }

  peerPrint (peer: peerType) {
    //console.log(".........................")

    let peersKeysList = Object.keys(this.peers[0])

    for (let i = 0; i < peersKeysList.length; i++) {
      console.log("peer[peersKeysList[" + i + "]]: ", peer[peersKeysList[i]])
    }
    //Object.keys(peer).forEach(key => {
      //console.log(peer[key])
    //})
  }

  random_id = () => { // it mimics hash function
    return Math.random().toString(36).substr(2);
  }

  syncPeerDocsFeed = (peer: peerType): void => {

    let peersKeysList = Object.keys(this.peers[0])

    let peer_docsListLength = peer[peersKeysList[1]].length;

    for (let j = 0; j < peer_docsListLength; j++) {
      this.syncStates[peer[peersKeysList[0]]][peer[peersKeysList[1]][j]] = this.random_id()
    }
  }
  initSync() {

      // https://github.com/automerge/automerge/blob/main/SYNC.md#connecting-decodesyncstate-or-initsyncstate
      // When a peer is discovered, first create a new syncState via initSyncState(),
      // and store the result somewhere associated with that peer.
      // All subsequent sync operations with that peer will return a new syncState to replace the previous one

    for (let i = 0; i < this.peers.length; i++) {
      this.syncPeerDocsFeed(this.peers[i])
    }

    console.log("syncStates : ", this.syncStates)
  }
}

export default AutomergeBackend

import AutomergeBackend from './AutomergeBackend'

type docsHandledByPeer = string[]

interface peerType {
  peerId: string,
  docs?: docsHandledByPeer,
}

let peers: peerType[] = [
  {
    peerId: "alice",
    docs: ["doc1", "doc2"]
  },
  {
    peerId: "bob",
    docs: ["doc1"]
  }
]

let automerge_backend = new AutomergeBackend(peers)
automerge_backend.initSync()

Executing it I get :

syncStates :  {
  alice: { doc1: 't2ulsx74d4', doc2: 'b48rqrvilp7' },
  bob: { doc1: 'dznzlmbcxk6' }
}

I also added updatePeers as class method:

  // https://github.com/automerge/automerge/blob/main/SYNC.md#synchronizing-with-one-or-more-peers
  updatePeers(docId: string) {
    Object.entries(this.syncStates).forEach(([peer, syncState]) => {
      const [nextSyncState, syncMessage] = Automerge.Backend.generateSyncMessage(
        this.backends[docId],
        syncState[docId] || Automerge.Backend.initSyncState(),
      )
      this.syncStates[peer] = { ...this.syncStates[peer], [docId]: nextSyncState }
    })
  }

Is, according to you, my implementation of syncStates reasonable and fully correct? I looked for clues about syncState interface but I didn't find any:

https://github.com/automerge/automerge/blob/d82f6208a485376fc07b18dda43af8dfa2fde7c2/%40types/automerge/index.d.ts#L201

How would you suggest me to implement backends ?

[raphael10-collab]

I've found this automerge-demo: https://github.com/pvh/automerge-demo I'm going to dive into it. And will come back

[pvh]

Ah good, @raphael10-collab. I'd seen your message but wasn't at my desk over the weekend. Feel free to ping me here or on the automerge Slack with questions and I'll try to help you along.

[raphael10-collab]

I'm trying to port this automerge-demo to a react-typescript environment.

As starting point, I just imported in a brand new react-typescript app the automerge-demo's files.

Typescript checking I'm getting these errors:

  [12:13:55 PM] Starting compilation in watch mode...

  src/automerge-store.ts:4:29 - error TS2307: Cannot find module './worker.ts?worker' or its corresponding type declarations.

  4 import AutomergeWorker from './worker.ts?worker'
                                ~~~~~~~~~~~~~~~~~~~~

  src/automerge-store.ts:5:31 - error TS2307: Cannot find module './shared-worker.ts?worker' or its corresponding type declarations.

  5 import PersistenceWorker from './shared-worker.ts?worker'
                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~

  src/automerge-store.ts:32:5 - error TS2552: Cannot find name 'update'. Did you mean 'Date'?

  32     update((doc) => {
         ~~~~~~

    node_modules/typescript/lib/lib.es5.d.ts:907:13
      907 declare var Date: DateConstructor;
                      ~~~~
      'Date' is declared here.

  src/automerge-store.ts:32:13 - error TS7006: Parameter 'doc' implicitly has an 'any' type.

  32     update((doc) => {
                 ~~~

  src/automerge-store.ts:50:14 - error TS2552: Cannot find name 'update'. Did you mean 'Date'?

  50       } else update((doc) => Frontend.applyPatch(doc, message.patch))
                  ~~~~~~

    node_modules/typescript/lib/lib.es5.d.ts:907:13
      907 declare var Date: DateConstructor;
                      ~~~~
      'Date' is declared here.

  src/automerge-store.ts:50:22 - error TS7006: Parameter 'doc' implicitly has an 'any' type.

  50       } else update((doc) => Frontend.applyPatch(doc, message.patch))
                          ~~~

  src/db.ts:19:32 - error TS2345: Argument of type 'string | null' is not assignable to parameter of type 'string'.
    Type 'null' is not assignable to type 'string'.

  19           db.deleteObjectStore(storeNames.item(i))
                                    ~~~~~~~~~~~~~~~~~~

  src/utils.ts:20:21 - error TS2683: 'this' implicitly has type 'any' because it does not have a type annotation.

  20     const context = this
                         ~~~~

    src/utils.ts:19:10
      19   return function (...args: any[]) {
                  ~~~~~~~~
      An outer value of 'this' is shadowed by this container.

  src/worker.ts:8:21 - error TS2304: Cannot find name 'WorkerGlobalScope'.

  8 declare const self: WorkerGlobalScope
                        ~~~~~~~~~~~~~~~~~

  [12:13:57 PM] Found 9 errors. Watching for file changes.

[corwin-of-amber]

I too want to try the new preview version of automerge 1.0.0, and would appreciate some simple working example. I used DocSet and Connection in the past so I'm familiar with the idea. What I want to see is a single snippet where you:

• Initialize a document (well that I already know)
• Initialize a (new) sync state
• Change the document
• Generate the sync messages corresponding to the change
• Apply the changes to a separate copy of the document.

Also, Connection used to know when changes occur in the document without the user having to initiate synchronization, and DocSet had a registerHandler which was most convenient for this purpose. Is there a replacement for this functionality?

[ept]

@raphael10-collab Hopefully @pvh can help you with the automerge-demo.

Hi @corwin-of-amber, you can look at the tests and the sync protocol docs for some self-contained examples.

Also, Connection used to know when changes occur in the document without the user having to initiate synchronization, and DocSet had a registerHandler which was most convenient for this purpose. Is there a replacement for this functionality?

Do you want to receive a callback whenever a document changes? You can use the Observable API for this.

[douira]

Does https://github.com/pvh/automerge-demo use the latest version of Automerge? I've also been unsure about which version all the other demo projects using Automerge were using and if the API they are all using is still the most current one.

[pvh]

Yes, automerge-demo is intended to show off how to use 1.0.

On Tue, Oct 12, 2021 at 4:49 AM douira @.***> wrote:

Does https://github.com/pvh/automerge-demo use the latest version of Automerge? I've also been unsure about which version all the other demo projects using Automerge were using and if the API they are all using is still the most current one.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/automerge/automerge/issues/437#issuecomment-940938845, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAWQC4FMPU22M7Q6AADELUGQOFDANCNFSM5FGL4XWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Peter van Hardenberg "Everything was beautiful, and nothing hurt."—Kurt Vonnegut

[corwin-of-amber]

Thanks @ept! These would be helpful pointers. I assumed the Observable API would come in handy from the name, so I will follow the examples in the tests.