Initiate Data Handshake on Reconnect

[PsychoLlama]

TL;DR: if you make changes when offline, they won't be propagated upon reconnection.

Gun version 0.3.998, the latest on npm.

Replicating the bug

In the examples folder, make some changes to allow other computers to connect:

todo/index.html

Naturally, localhost won't work from another computer.

// Line #14
-var gun = Gun(location.origin + '/gun');
+var gun = Gun('http://your-ip-address:8080/gun')

http.js

Allows other computers to connect via IP address.

// Line #28
-server.listen(port, ip);
+server.listen(port);

1. Open http://you-server-ip-address:8080/todo/index.html on two computers.
2. Add some todos, they should sync.
3. Take one computer offline and add some todos.
4. Take it back online and add more todos.
   You should notice that only the todos created while online were synced in real-time.
5. Refresh that browser, and the offline todos should (finally) replicate.

Props to @nevf for finding this bug!

[nevf]

@PsychoLlama Also note that todo's deleted while offline don't sync when back online.

[PsychoLlama]

@nevf thankfully that's part of the same problem, so both should be fixed at once :sweat_smile:

Deletes in gun work by writing null to a field, meaning it operates on the same mechanics as writes. Fixing one should fix the other!

[PsychoLlama]

So in some cases, the websocket readyState isn't updated when the client goes offline. It seems the platform/browser combo I was testing on (chrome v54 on android) was just gracious. The next solution that comes to mind is waiting for a server acknowledgment, but the websocket server doesn't give one (that I can tell).

[amark]

sounds like this was a great fix! Closing since the pull got merged. Please reopen if anything weird is going on.

[nevf]

@PsychoLlama @amark I've just got around to testing the latest master release "0.3.9991" and still am seeing much the same issues.

1. Start server on PC 1
2. Open Browser on PC 1 and on PC 2
3. Disable LAN on PC 1
4. Add 2 todo items on PC 1 and PC 2
5. Enable LAN on PC 1
6. PC 2 updates and shows all 4 todo items. PC1 only shows the items added therein.

I've also had issues where deleting a todo item when the LAN was disabled, which didn't update the other PC.

I'm using Win 10 on both PC's. PC 1 is on a Wired Network and PC 2 is connected via. WiFi. Chrome Version 54.0.2840.71 m

[amark]

@nevf, thanks. I had tried a similar variation the other day and it worked. We'll try to replicate and get back to you.

What's your project again? :)

[nevf]

@amark I'm not using Gun yet, just an interested bystander at this point. That said I've done a lot of database work including pushing updates out in real time using Websockets to Browsers in my Knowledge Base web app Clibu

Right now I'm especially interested in offline first, and the ability to use a fully fledged database app offline and then sync (resolving conflicts) when back online again. Which is part of the reason for testing gun in this scenario.

[PsychoLlama]

Hey @nevf, sorry not trying to hijack the conversation... in relation to the offline queuing issue, @amark helped me figure out how the websocket request/response model works in the browser, hopefully I can get a more robust fix out soon :grinning:

[nevf]

@PsychoLlama Great, look forward to giving it another work-out. FYI I've always been good at breaking things. ;-)

[PsychoLlama]

Haha, awesome @nevf! We need more users who break things, it helps us find the bugs :D

[amark]

@PsychoLlama didn't you try testing this again and everything worked fine?

[PsychoLlama]

@amark yeah, which is odd, since it was failing for me just a bit ago, then it started working again magically and I can't re-create. Running the same gun version, and the same operating system versions (aside from a minor update to windows). Next step is to delete all the gun files, re-install and try again fresh.

[PsychoLlama]

@amark! While working on PR #292, I think I figured out what's going on, and offline queues only solve half the problem.

Two things:

• Our browser sockets are lazy. They'll only (re)connect if something needs to be sent.
• When a reconnection is made, there's no handshake.

That creates a pretty bad combo:

1. The server goes offline for a minute, closing the sockets it had with two browsers, Alice and Bob.
2. Bob sends a new message, forcing his socket open again.
3. The server sees the message and forwards it to other connected clients, but since Alice hasn't sent a message yet, her socket is still closed. She never sees Bob's update.
4. Handshakes (currently) only happen the first time data is requested. Once Alice eventually does reconnect, no handshake happens. She'll only see updates sent after the reconnection. When she reloads the page, it forces a new handshake, loading all the updates she missed before.

I think that's what's causing our issue.

A possible solution is to automatically attempt a reconnect from the browsers. That narrows the space where we might miss an update, but at least it's an improvement. Since messaging is managed by gun, the handshake is probably the more complex of the two.

Thoughts, @amark?

[PsychoLlama]

Per an IRL conversation with @amark, we're automatically attempting a reconnect after socket disconnect in PR #292. Although it narrows the window for missed updates, it still leaves the handshake logic undealt with, so we're gonna leave this issue open for now.

Note: If the browser is refreshed, all the missed messages will show up.

[amark]

We got this mostly fixed, but going to change the title to reflect the more recent things that are left still to do. Keeping this open.

[gordongordon]

hi, Amark and Jesse I am new to Gundb, but having been playing for a month now! I am wondering, does it work for the current version of gundb, 0.7.4! Since, I am writing a collaborative todo website with React and Mobx and Gundb, but I get the same problem which was mentioned before!

How is happen?

1. Firstly, start User1 Browser, start User2 Browser, start User3 Browser and server offline!
2. Secondly, user1 adds a todoA record, user2 adds a todoB record, user3 adds a todoC record, then server goes online
3. Thirdly, user1 got updated todoA, TodoB, TodoC, user2 got updated todoA, todoB, user3 got notthing!

Note: I did try some of demo with the version of gundb 0.7.4. and it works fine when server is online!

Here is the client's Gundb setting:

var gun = Gun(
                'http://127.0.0.1:8080/gun' : {
                backoff: { time: 100, max: 5000, factor: 0.2 }
              }
  );

Here is my server's Gundb Setting:

var gun = Gun({
    file: 'data.json', // Saves all data to `data.json`.
    web: server // Handles real-time requests and updates.
});

will look forward to hear from you! thx!

[amark]

@gordongordon thank you for reporting this! I'll set up a PANIC test to verify and test against. So we can get it fixed. Just to double check:

User 1 - 3 load the page, and THEN we turn the server off. Correct? (or does User 3 join AFTER the server is offline?)

Is it /always/ the 3rd (last) user that fails to get the updates on reconnect? Or is it just 1 of the 3?

[gordongordon]

@amark User 1 - 3 load the page, and THEN we turn the server off. Correct? Yes, Does User 3 join AFTER the server is offline? I would say Yes, since User 1 ~ 3 always online. As I test my app today and the results are not the same most of time. Which means sometime work , some time doesn't! I think, doesn't matter I add todo, or I remove todo. Thx look into that! Because, I am serious about since, I use it in a production application.

[amark]

@gordongordon great! We'll get this fixed for you. I'm in SF right now, but will have a test written for it next week and then hopefully a fix soon after that. This is obviously a pretty serious bug! So thank you for complaining about it.

Would love to hear more about your app in the mean time!

[gordongordon]

@amark Thx lots! Will keep in touch with you! I am developing a website by using react and mobx and gundb, the website is for users to post a property request such as to buy, to sell , as a result each user is listening on e.g. property.get('forSell').map().on( .. ) . for an update! It is base on a todo application from the beginning.
I would say, a matching application with real time status updating for properties. Sure, by using graph database design in relationship of buyers, sellers and agents!

[amark]

@gordongordon it took me a while longer to be able to focus on code, but I believe the most recent update v0.7.9 should now fix your problem. Could you please upgrade and test your app to see if it works now? Thanks, let me know!

[gordongordon]

@amark Yes, thx! Will run a test this weekend! And lets u know the result!

[amark]

https://github.com/amark/gun/issues/392

https://github.com/amark/gun/issues/499

[odama626]

Been thinking about using Gun in production for a project. Is this still an issue with the latest version?

[odama626]

Been thinking about using Gun in production for a project. Is this still an issue with the latest version?

[nevf]

+1 I'd also like to know if this has been fixed. Been a long time in the queue.

[amark]

Yes if using localStorage (on by default).

No if not.

I hope this helps with the decision process!

I'd love to catch up with both of you and hear about what you are working on.

[amark]

Also @omarzion & @nevf looks like @rogowski also already coded a non-localStorage dependent solution in the experimental AXE branch https://github.com/amark/gun/pull/648 !

[alfonsocuccaro]

Hi, I'm exploring Gun for an enterprise application (a Point Of Sale) and it really looks great, but this issue is pretty important for my requisites. I have tried in all possible ways and it looks like when a peer goes offline never sends old data after reconnection. The only possible way to get that data is to 'on' that data from another peer but only after you refresh the page of the second peer (very strange), also using localStorage.

I have tested it on the Todo Example at https://github.com/amark/gun/blob/master/examples/todo/index.html connecting Gun instance to a local relay server, here the code

const GUN = require('gun');
const http = require('http');

const server = http.createServer().listen(8080);

gun = GUN({
    web: server,
    file: 'file',
});

and I have tested it also on existing Gun projects found in the documentation (e.g. https://gun-next.example.allthethings.win ) and when a peer goes offline it loses the sync.

I have 2 questions:

1. How can I resolve this situation?
2. If it is expected to happen, how can I workaround it and "force" the sync?

[clibu]

@alfonsocuccaro I suggest you have a close look at YJS which I'm using in a new app and it is amazingly good, close to magic. Also on Github and Discord etc.

It is disappointing that after all these years this most basic of issue seems to still not have been addressed. Also I see Gun now has 213 Open issues, which doesn't inspire confidence.

[odama626]

@clibu to add to your comment, I had some minor successes back in 2017 with gun but I've found some major breaking bugs and opened issues that just sit open for years, I don't have faith in building anything seriously with this project, even though the idea of it is fantastic and I would love it to work.... I've even dug through the codebase to see if I could try and help contribute, but it looks like it is being written in a minified way almost to make it more obtuse.

Thanks for the YJS suggestion, I'll check it out

[clibu]

@odama626 Yes when Gun first came out I also thought it was interesting and potentially useful. However I was always concerned about it being called a Database and was written because of failings with MongoDb. Unless things have changed Gun is an in-memory graph which somehow, sometimes synchronizes across devices and has backing stores to persist memory to disk. I don't consider to be a "real database".

I see almost no discussion of Gun in Distributed / Decentralized Database, CRDT and the like discussions. The Flashy Web site looks aimed at hobbyists and hackers which is fine, however it is hard to see it being taken seriously for real world business use.

You would think with "60+ million monthly downloads" and funding, that issues would be resolved promptly.

I forget to mentioned they great work the tiptap folks are doing with https://tiptap.dev/hocuspocus which gets a backend for YJS up and running quickly.

Disclaimer I am sponsoring Hocuspocus and YJS.

[alfonsocuccaro]

@alfonsocuccaro I suggest you have a close look at YJS which I'm using in a new app and it is amazingly good, close to magic. Also on Github and Discord etc.

It is disappointing that after all these years this most basic of issue seems to still not have been addressed. Also I see Gun now has 213 Open issues, which doesn't inspire confidence.

YJS is really great. That's what Gun should do. If someone is reading this post because you can't understand what Gun says go to YJS and you will solve all of your problems.

Thank you @clibu

[bart]

@amark I am currently testing Gun JS and it looks like the re-sync issue still exists. Having two separate browser open connected to a peer instance. Wen peer instance goes down, a client changes data and the peer comes back online no data is synced. As far as understood your Youtube talks as well as the documentation, that should work, right? Do you have any updates on this? Thanks!

[amark]

YJS is a great CRDT, but (some of you), are absolutely distasteful.

I was a single dad, raising 2 kids, broke, and providing free support and open source tooling to people. My case is going to the supreme court, my kids got locked up by someone who abused me.

Yes, I prioritize the non-profit/ethical-startups, because scaling their 60M users is non-trivial to workaround.

Yet [person(s)]'s response is entitlement. Entitlement entitlement entitlement. "I pay an Open Source maintainers $0 and if they don't help me I'll trash falsisms on their project because that's how much of a pathetic narcissist I am."

I'm sorry a good project, like YJS, will inherit toxic people/person(s).

For those of you who still need help, or can't use YJS because you need mutable data or performance/scale, please use http://chat.gun.eco, we've helped thousands of people there.

If someone wants to help me instead, volunteering to answer the hundreds of question issues & closing them, and writing tests for the ones that remain as bugs, would be deeply appreciated.

[alfonsocuccaro]

@amark I'm sorry for your story and I don't want to be offensive, but this feature is not optional for a system like GunJS and I have tried for almost a month to use it for an enterprise project but it is uncomplete and (currently) useless. I wrote more than one message on gun's chat and never got a solution. Morover I haven't found one single active project using gun. The projects list (https://gun.eco/docs/Awesome-GUN) is made of broken links, non-working projects, and sometimes-working projects (because of the problem exposed in this issue).

So, I don't want to appear too critic, but I don't think I said falsism on your project. That said, you are free to do as you wish, but I think contributors will help you with your project if it is uncomplete, but at least works.

Good luck

[draeder]

@alfonsocuccaro Why not edit the awesome-gun wiki to fix the broken links, and notate the non-working projects instead of complaining about it?

[alfonsocuccaro]

@draeder 1) to fix the broken links I should remove the links because the projects don't exist anymore (courious they don't exist anymore, ask why) 2) you can't extract a single point from a list to accuse me 😂😂😂

Anyway at this point I have two hypothesis: you can't understand or you don't want to understand the real point of the entire situation. In both cases I can't do anything more so do whatever you want, I say this project will remain a didactic experiment forever , but maybe you know a lot of things that I can't understand.

Bye

[amark]

@alfonsocuccaro your falsism:

"""We're an enterprise that somehow can only afford to bash an open source project if its lead maintainer with 60M monthly users doesn't help us for free instantly."""

Sure, "enterprise".

Enjoy failing at what "hobbyists" highschool students have done trivially. Let me know when you're bigger than Internet Archive.

[alfonsocuccaro]

You are right, I'm stupid and @amark is a genius 👏🏻👏🏻👏🏻

"Offline first database" --> works perfectly only if you are online. Good job 👍🏻