search

derekkraan / horde

Horde is a distributed Supervisor and Registry backed by DeltaCrdt
MIT License
1.3k stars 102 forks source link

Occasional Horde Registry Lookup Misses #247

Closed Miserlou closed 2 years ago

Miserlou commented 2 years ago

I'm having an unfortunate occasional issue where a new game gets registered using Registry, a PID is initially returned, but upon looking up the PID using Horde.Registry.lookup(MyApp.GameRegistry, game_code), it returns nothing. I'd say this happens for about 1 in 10 new games. I can see that the clustering is working fine otherwise from Node.list.

The issue is very similar to the one described here: https://github.com/derekkraan/horde/issues/225 - but the solution described there did not help my problem.

It's quite frustrating, as it isn't deterministic, but happens enough to be a problem for the application. It goes away if I reduce the cluster down to a single node.

Any ideas what could be happening here, or if there's a way to make Horde more verbose to show what could be going on?

Miserlou commented 2 years ago

I think I've addressed this with delta_crdt_options: [sync_interval: 15].

derekkraan commented 2 years ago

Hi @Miserlou , Horde is eventually consistent, so there will always be a short delay (governed more or less by the sync_interval as you have already discovered) before registered information is available in the wider system.

Note that a sync_interval of 15ms is quite low, it means your nodes will be syncing 66 times a second.