Support offline and sync

[delaneyj]

Watching a talk from @WarrenBuffering at BigSkyConf. He showed a Reactive Native app that I thought should be available in Datastar. Have working example and it's pretty simple.

Added a data-store.offline as a modifier. This will auto save on updates and will load from local store on startup.

The other is the data-on-* can have a .window modifier. Otherwise, this "just" works.

<div
     data-store.offline='{"stuffAlreadyInStore":"this will be in the store even without network"}'
     data-on-online.window="console.log('online, syncing'); $$put('/examples/offline_sync/sync', false)"
     data-on-offline.window="console.log('offline')"
>
     <div id="results">
</div>

Built off inspector branch so will take a bit to make mainline.

[gedw99]

wow this is really amazing functionality @delaneyj

I am quite curious what it syncs ? Is it data like json or is it page fragments ?

[Superpat]

@gedw99 on-online.window is just an event so the $$put it does is just a standard datastar SSE request. It sends up a json formatted store and you send down whatever you want.

[delaneyj]

Yep, talking with Warren I thought it would be straight forward. Didn't think it's be 2 minor modifiers to existing attributes. Just nice when it works out that way.

[gedw99]

So then we can do things like ....

so we are syncing JSON between the client and server.

so for a Form, we dont need a traditional HTML form anymore. We can just move the JSON over the SSE transport.

so because it's SSE, then we could then also rather easily sync it to other client when the JSON changes.

[delaneyj]

In datastar (unless you are doing large file uploads) you never need forms. Datastar already sends off all signals to the server on the $$post like actions. Since anything reactive should be in signals its very easy to sync state. That and the data-on-* plugin supports any addListenerEvents it was very straightforward.

[gedw99]

I noticed that from the code and debugging . It’s huge development productivity gain.

with a user concurrently using many tabs , would it keep their tabs in sync ?

[delaneyj]

Yes, that's what happens automatically with Datastar

[gedw99]

Wow, that is a hugely productive thing for a User and of course a Developer that wants this architectural feature for free.

I just tried it with: https://data-star.dev/examples/click_to_edit

The code being: https://github.com/delaneyj/datastar/blob/main/backends/go/site/routes_examples_click_to_edit.templ

When I change the First Name in Tab 01, in Tab 02 I don't see the First Name change unless I refresh. This seems like a decent example of where sync would be beneficial because Tab 02 would update a few milliseconds after Tab 01 First Name is changed.

---

The same is true for https://data-star.dev/examples/delete_row

The code being: https://github.com/delaneyj/datastar/blob/main/backends/go/site/routes_examples_delete_row.templ

In Tab 01, if I delete a row, Tab 02, would delete the respective row. At the moment it does not work for me, but I guess it's because the Production version is too old.

I need to update my local code again and have a look...

[delaneyj]

Offline is on a different branch, but it'd still be an opt in feature in mainline. Caching is always a hard problem and I'd default to it being off

[Superpat]

Well thats because the example doesnt have an ongoing Get request doing the syncing. If you add an on-load to a server route that polls for changes to that value, then it would work.

-------- Courriel d’origine -------- De : ged @.> Envoyé : 13 juin 2024 17 h 59 min 48 s HAE À : delaneyj/datastar @.> Cc : Patrick Marchand @.>, Comment @.> Objet : Re: [delaneyj/datastar] Support offline and sync (Issue #74)

Wow, that is a hugely productive thing for a User and of course a Developer that wants this architectural feature for free.

I just tried it with: https://data-star.dev/examples/click_to_edit

The code being: https://github.com/delaneyj/datastar/blob/main/backends/go/site/routes_examples_click_to_edit.templ

When I change the First Name in Tab 01, in Tab 02 I don't see the First Name change unless I refresh. This seems like a decent example of where sync would be beneficial because Tab 02 would update a few milliseconds after Tab 01 First Name is changed.

---

The same is true for https://data-star.dev/examples/delete_row

The code being: https://github.com/delaneyj/datastar/blob/main/backends/go/site/routes_examples_delete_row.templ

In Tab 01, if I delete a row, Tab 02, would delete the respective row. At the moment it does not work for me, but I guess it's because the Production version is too old.

I need to update my local code again and have a look...

[gedw99]

Thanks @delaneyj and @Superpat

wow this is crazy cool.

yeah I just saw the Offline branch. Ok I will see how far I can get based on your tips.

[gedw99]

Thinking about the best way to do this.

Corrosion or SurrealDB both support DB feeds. Corrosion is very easy to install and run.

https://github.com/superfly/corrosion

https://superfly.github.io/corrosion/api/subscriptions.html

golang sync code: https://gist.github.com/jeromegn/22d88b1b0399b40d4a714b4367dec430

The DB is SQLITE with a CRDT system to allow it to be master master on the servers and to do query subscription, such that a change to a row in the DB, causes a http call to the middle tier with the exact nature of the change ( which is what we want ) , and so then allowing a sync to the browser to then occur.

All Datasra servers in your cluster will get the change event firing from Corrosion, so all users browsers will sync.

[delaneyj]

Corrosion looks crazy complicated versus my NATS+SQLite since its pure Go and a single binary for both app and servers. You don't need CRDT in SQLite if you use jetstream as your source of truth. For stuff that would use Datastar's offline sync I'd be surprised if you need more than just NATS KV. I'm gonna change out the front page with a todomvc that supports both offline/local and global todos. That will only use natskv

[Superpat]

For postgres I guess you could use something like NOTIFY

-------- Courriel d’origine -------- De : ged @.> Envoyé : 13 juin 2024 21 h 05 min 49 s HAE À : delaneyj/datastar @.> Cc : Patrick Marchand @.>, Mention @.> Objet : Re: [delaneyj/datastar] Support offline and sync (Issue #74)

Thinking about the best way to do this.

Corrosion or SurrealDB both support DB feeds. Corrosion is very easy to install and run.

https://github.com/superfly/corrosion

The DB is SQLITE with a CRDT system to allow it to be master master on the server and to do query subscription, such that a change to a row in the DB, causes a http call to the middle tier with the exact nature of the change ( which is what we want ) , and so then allowing a sync to the browser to then occur.

[gedw99]

@delaneyj

Yeah there is also a pure goalng approach with SQLite and Marmot ( which is NATS Jetstream ).

https://github.com/maxpert/marmot

It's multi-master and eventually consistent thanks to NATS. SO there is no locking. It applies the LWW rule.

This means there is NO serializability guarantee of a transaction spanning multiple tables. Same as Corrosion also I believe.

Pocketbase uses it to scale out. It's very easy to setup and run.

I got it running on fly in 8 different data centres, with NATS embedded in Marmot. https://github.com/maxpert/marmot-pocketbase-flyio

Its worked fist time :)

[gedw99]

Corrosion gives you a nice ability to do SQL Subscription Queries.

Surreal also allows you to do Subscription Queries.

Its possible to maybe do it with Marmot too, but not sure.

[delaneyj]

I think most real world SaaS aren't going to need to scale beyond NATS+SQLite. But again, this has little to do with Datastar and is orthogonal concerns.

[gedw99]

nats and sqlite works really well for me with marmot

[gedw99]

could you add an example that does have the sync stuff one day ?

Just one where each h browser tabs stays in sync, would be fab.

[delaneyj]

The new https://github.com/delaneyj/datastar/tree/todo-example branch should work without any sync cross tabs cause it's using a session cookie.

[gedw99]

The new https://github.com/delaneyj/datastar/tree/todo-example branch should work without any sync cross tabs cause it's using a session cookie.

Just so I understand, your saying that is DOES NOT do Browser Tab syncing ? I thought the idea was to add an example showing off Browser Tab syncing ?

I see github.com/gorilla/sessions was added, so then this is a state system that is used instead of using NATS ? is that the intent ?

BTW thank you for adding this... Dont mean to sound ungrateful.

[delaneyj]

Just so I understand, your saying that is DOES NOT do Browser Tab syncing ? I thought the idea was to add an example showing off Browser Tab syncing ?

Just tried on the new todo-example branch. If you browser tabs share the same session they get sync'd without the need for any sync plugin. Because the example uses a single GET and uses NATS in a CQRS fashion it updates correctly from either tab.

1. The store is to have cookies to track a session id
2. that then is the key in the kv for the state.
3. Each browser tab is watching their key for current state
4. ... you're done, there is no extra steps, every staying in sync from anywhere (that shares the same key).

You could have an OAuth flow that then you create a cookie from but the same process applies. In this case I TTL the kv and limit to a max bytes in case of DDOS

[gedw99]

Thankyou @delaneyj for the explanation and making this.

It really sets up for powerful use cases.

[delaneyj]

@gedw99 check out the front page for the todo example using NATS kv watchers.

[gedw99]

Thanks 🙏

It works for me on Safari syncing between tabs.

There is a bug with pressing the checkbox. Have to press twice if not already focused on the checkbox.

Same goes for the text input box. It’s 3 presses to go from 1 to the other.

I guess the optimistic way to do this that when a user presses on text , then assume all other should be changed to text , and to move the focus into the text input box ?

It’s pretty cool though how the checkbox is synced too though.