Unable to get consistent reads with GUN

[Azarattum]

Callback from .on or .once sometimes just doesn't fire without a sane reason. I was able to recreate a minimal reproduction.

Reproduction

Server

const GUN = require("gun");                     //npm i gun
const { createServer } = require("http");

const gun = GUN({ web: createServer().listen(8765) });
const db = gun.get("db");

// Run this once to save the data
// db.get("data").put("somedata");
setInterval(() => {
  db.get("test").put(Math.random().toString());
}, 1000);

Here we create a simple HTTP server, put some date in db.data and make an update cycle for db.test. Note, that we put "somedata" in db.data than comment it out to use value from a disk.

Client

JSBin

localStorage.clear();

gun = GUN("http://localhost:8765/gun");
const db = gun.get("db");

db.get("data").on((x) => console.log(x));
db.get("test").on((x) => console.log(x));

Here we listen on updates in both db.data and db.test.

Results

We expect to see the following in the client's console.

"somedata"
"0.9583885600760855"
"0.7716893870606449"
...

And this is what we get for the first time when we visit the page (everything works as expected). However, if we refresh the page (or rerun a JSBin) we get:

"0.9200648211947335"
"0.8705901553854474"
...

db.data callback never fires! This is true for the following update. The only way to fix this is to restart the server. So,

We get db.data back only from the first connection to the server!

When

we receive db.data:

• on the first connection after server restart
• on every connection if we uncomment db.get("data").put("somedata");
• on every connection if we comment out setInterval

we don't receive db.data: <- BUG

• after the first connection if setInterval is present and the data is coming from disk (no put)

Network

First connection network logs (received "somedata")

Second connection network logs (nothing changed on the server, client refreshed, localStorage on the client disabled, didn't receive db.data)

Impact

Critical. Core functionality doesn't work as expected. Let me know if there is a workaround. Without one GUN isn't very usable :(

[Azarattum]

Video Demonstration

https://user-images.githubusercontent.com/43073346/183663305-7d565cc8-49b0-4a98-aa43-9d87af1563ef.mov

[Azarattum]

The described above was using the NPM version. Just tested on the latest git version, the issue persists...

[Azarattum]

If we subscribe to db.data on the server, the data is sent as expected.

db.get("data").on((x) => console.log(x));

[Azarattum]

If we subscribe to db.data on the server, the data is sent as expected.

db.get("data").on((x) => console.log(x));

I found a simpler hack using db.once(); on the server. For some reason gun.once(); did not work.

[amark]

This is one of the best bug reports I've seen, thank you @Azarattum .

Definitely higher priority need to get fixed - usually this is just sitting down and forcing time on it. Would you schedule a call with me ASAP? Fastest: DM me on twitter/marknadal.

One of the first things we'll check is GUN({axe: false in relay. If bug still persists, then means its a core GUN issue (well, possibly RAD?) which makes it easier to find where to fix, knowing which component it is in.

Thanks again. Shoot me some times (idk why, but github doesn't notify me anymore, so DM please if you can!)

[draeder]

@Azarattum Mark and I may have just fixed this after identifying a bug and issuing an emergency patch. Please npm i gun@latest, test again and let us know! Fingers crossed

[Azarattum]

@draeder, @amark thanks for the quick response! @draeder I'm afraid the issue still remains on the latest version. I tested both with axe: false and axe: true. @amark if still necessary, I'll DM you on twitter later to schedule the call. I'm quite busy right now, but I think I'll manage to find some time. My time zone is GMT+07:00.

Video Demonstration

https://user-images.githubusercontent.com/43073346/183820504-cc763dfb-6f88-47ea-b0bd-ffdb6f59de23.mov

[Azarattum]

Actually now it seems even worse. The callback for db.data now occasionally fires twice from the first hit. And still consistently doesn't fire on any further hits.

https://user-images.githubusercontent.com/43073346/183825726-289804d3-c275-4186-8439-27f1c188c1b7.mov

[draeder]

@Azarattum No bueno! @amark and I tend to pair program around 3/4pm CST .. Perhaps you can join today depending on @amark 's schedule?

[draeder]

@Azarattum I haven't been able to reproduce the original issue locally on the new version, in node only, nor with node + jsbin (perhaps a browser cache issue?). EDIT: Nevermind --> I forgot to remark out the 'some data' put.

I do, however, get the initial 'some data' twice.

A workaround for that:

let last
db.get("data").on((x) => {
  if(x === last) return
  console.log(x)
  last = x
});

[amark]

@Azarattum can you join our call today? Sometime 1pm+ PST? Remember, DM me twitter/marknadal

[amark]

we did a call.

where the bug, is identified

(RAD adds itself to an in-memory cache, but GUN doesn't know about this - so then both think the other will take care of it, as a result, neither do)

I need to decide if I need to fix GUN or RAD.

Will keep you all updated.

[amark]

@draeder on one of our next calls, let's figure out where this belongs: RAD, GUN, etc.

[Azarattum]

Hey! Any progress on this?

[draeder]

We should be able to tackle this some time after this week.

[amark]

Sorry I got swamped with traveling/conferences the whole last month. Now my laptop is missing... tho @draeder maybe this is something we can decide on (or Norman helps me with it in RAD) that can do as a isolated branch, without my uncommitted missing-laptop changes?

[amark]

ugh. this still hasn't been fixed. help please.

[Jahb]

@Azarattum @amark Were you able to find a workaround/bandaid for this? I think I'm getting something similar but I haven't set up a demo/sandbox for it yet.

[amark]

@Jahb replicating is easy now, per @Azarattum 's excellent work. I've been suffering a million other cuts, but this would be good for all if fixed - anyone want to hunker down on it with me on a screencall?

[0awful]

Is this issue still present?

[amark]

this is a pretty bad bug.

[Azarattum]

Any updates on this?

[mblaney]

Hi I've just opened pr https://github.com/amark/gun/pull/1371 that should fix the missing reads if you want to test it out?

From what I can see the tmp.rad timestamp is set for the soul, db in this case, which is getting updated due to the test puts, but misses the fact that data was requested. This pr just checks the graph for the atom requested instead, which I'm not sure is the best solution, but happy to discuss if there's a better place to fix this.

[amark]

HOLY COW @mblaney you are a HERO (see https://github.com/amark/gun/pull/1371#issuecomment-2037850720 ) gonna repeat comment again tho... this looks extremely similar to the branched fix @draeder & I had inside GUN code that I wasn't comfortable with because (A) didn't know cause edge cases (B) GUN not suppose to know about RAD. This is so obvious, wish I had thought about it sooner. Please see above PR comment as would like to move into an emergency (year+ late, sigh) NPM publish.

[amark]

Excited to say this bug is fixed thanks to @mblaney ( https://github.com/amark/gun/pull/1371 )! Sorry @Azarattum I didn't think of this approach, so obvious once seen, and shouldn't cause any side-effects that I was worried about in my attempts.

ANNOUNCEMENT: emergency (year+ late) release published to NPM, gun@v0.2020.1240 now available. PLEASE update your GUN relays IF you installed them via NPM. Note: That NPM release contains 0 of the last year's worth of fixes in GitHub main/master, because some of those changes are "major" version changes. So if you're running/installing from GitHub (and not having issues) keep doing that, as it also includes the fix.

NOTE: Old relays do not have auto version update, so most relays will still have this bug. I'm gonna try to hunt down my random providers and update mine though.

Closing!!! 🎉 🎊 🌟 🎉 🎊 🌟 🎉 🎊 🌟 🎉 🎊 🌟 🎉 🎊 🌟