cabol / nebulex

In-memory and distributed caching toolkit for Elixir.
https://hexdocs.pm/nebulex
MIT License
1.24k stars 74 forks source link

When a new node joins, the cache is not copied (Replicated) #231

Open cospin opened 3 weeks ago

cospin commented 3 weeks ago

Hello, I am new to Nebulex and I'm testing the Replicated adapter, but I notice that when a new node joins, it does not copy the cache from the other node. In production, DNSCluster is used to connect the nodes dynamically, but in local tests, I'm simply using Node.connect/1.

This is my setup:

defmodule Core.ReplicatedCache do
  use Nebulex.Cache,
    otp_app: :core,
    adapter: Nebulex.Adapters.Replicated
end
config :core, Core.ReplicatedCache,
  primary: [
    gc_interval: :timer.hours(12),
    gc_memory_check_interval: :timer.seconds(10),
    max_size: 1_000_000,
    allocated_memory: 2_000_000_000
  ]
def start(_type, _args) do
    children = [
      Core.Repo,
      {DNSCluster, query: Application.get_env(:core, :dns_cluster_query) || :ignore},
      {Phoenix.PubSub, name: Core.PubSub},
      {Core.ReplicatedCache, []},
      Core.Telemetry,
      Core.Endpoint
    ]
    opts = [strategy: :one_for_one, name: Core.Supervisor]
    Supervisor.start_link(children, opts)
end

I'm starting one local node and creating a new cache record:

iex --name node1@127.0.0.1 --cookie mycookie -S mix

Core.ReplicatedCache.put("key", "value2")
:ok

Then I join a second node, where it can't access the already created key:

iex --name node2@127.0.0.1 --cookie mycookie -S mix

Node.connect(:"node1@127.0.0.1")
true

# key not found, so no cache copy?
Core.ReplicatedCache.get("key")
nil

But new writes works fine between the two:

# put newkey from any node
Core.ReplicatedCache.put("newkey", "value2")
:ok

# get works on both nodes:
Core.ReplicatedCache.get("newkey")
"value2"

So, my conclusion is that when a new node joins, it is not copying the cache. Am I overlooking something, or am I doing something wrong?

Thanks!

Elixir 1.17.2 (compiled with Erlang/OTP 27)

cospin commented 3 weeks ago

Ok, I did a local test but this time adding libcluster, and it seems to work well :) I guess the problem was my rudimentary use of the cluster with Node.connect/1 (I'm also new to clustering).

I didn't see any of this in the documentation, so, pretty sure I'm doing something wrong with the cluster formation. Any insights? đŸ˜… Anyways, seems like libcluster is a nice addon.