adrienmo / eredis_cluster

eredis_cluster is an erlang wrapper for eredis to support cluster mode of redis 3.0.0+
MIT License
79 stars 83 forks source link

Infinity loop on pool max overflow #43

Closed alexandremcosta closed 4 years ago

alexandremcosta commented 4 years ago

If we reach a pool max overflow limit, or if we configure pool_max_overflow with 0:

  1. Client calls :eredis_cluster.q(["GET", "something"])
  2. Which calls :eredis_cluster.query/3 on src/eredis_cluster.erl:180
  3. Which calls :eredis_cluster_pool:transaction/2 on src/eredis_cluster.erl:186
  4. Which has this code:
    transaction(PoolName, Transaction) ->
    try
        poolboy:transaction(PoolName, Transaction)
    catch
        exit:_ ->
            {error, no_connection}
    end.
  5. And returns {error, no_connection} because we are out of poolboy workers
  6. Then, :eredis_cluster.handle_transaction_result/2 patterns match the result
  7. Which calls eredis_cluster_monitor:refresh_mapping(Version)
  8. Which calls :eredis_cluster_monitor.reload_slots_map/1
  9. Which calls :eredis_cluster_monitor.get_cluster_slots/1
  10. Which calls :eredis:q(Connection, ["CLUSTER", "SLOTS"])
  11. Which goes back to STEP 1, and starts the infinity loop

This triggers infinity error like:

(ErlangError) Erlang error: {:timeout, {:gen_server, :call, [:eredis_cluster_monitor, {:reload_slots_map, 1}]}}

Until we restart our application, which releases the pool of workers and our system is back...

alexandremcosta commented 4 years ago

Actually this is wrong, because the ["CLUSTER", "SLOTS"] call is done by eredis not eredis_cluster, so it does not trigger an infinity loop.