System memory spike caused by waiter_pids

[edescourtis]

Under a modest load of 100 calls per second for few hours, waiter_pids grows to more than 1MB in size. Because this process gets slower and slower the system eventually chokes up and memory grows rapidly until the system falls apart.

State: Scheduled
Spawned as: erlang:apply/2
Current call: 'Elixir.Memoize.Cache':compare_and_swap/3
Spawned by: <0.2426.0>
Started: Sun Oct 28 16:56:07 2018
Message queue length: 0
Number of heap fragments: 0
Heap fragment data: 0
Link list: [{from,<0.2426.0>,#Ref<0.1269697927.3293315074.117151>}]
Dictionary: [{rand_seed,{#{bits=>58,jump=>#Fun<rand.8.15449617>,next=>#Fun<rand.5.15449617>,type=>exrop,uniform=>#Fun<rand.6.15449617>,uniform_n=>#Fun<rand.7.15449617>,weak_low_bits=>1},[58928435655212311|10002483604556903]}}]
Reductions: 2218492
Stack+heap: 833026
OldHeap: 833026
Heap unused: 574022
OldHeap unused: 832833
BinVHeap: 0
OldBinVHeap: 29
BinVHeap unused: 46422
OldBinVHeap unused: 46393
Memory: 13329424
Stack dump:
Program counter: 0x00007f7cdddd1108 ('Elixir.Memoize.Cache':compare_and_swap/3 + 24)
CP: 0x00007f7cdddd17b8 ('Elixir.Memoize.Cache':do_get_or_run/3 + 536)
arity = 3
   {'Elixir.SmsSmpp.EsmeSession',get_supported_encodings,[<0.23941.302>]}
   {{'Elixir.SmsSmpp.EsmeSession',get_supported_encodings,[<0.23941.302>]},{running,<0.25628.302>,[<0.26802.302>,<0.25969.302>,<0.26610.302>,<0.29564.302>,<0.27613.302>,<0.26344.302>,<0.26388.302>,<0.26388.302>,<0.26748.302>,<0.26748.302>,<0.28154.302>,<0.26069.302>,<0.27488.302>,<0.27488.302>,<0.27038.302>,<0.27038.302>,<0.26460.302>,<0.26460.302>, ...
many many many many pids later
]}}

[melpon]

Thank you for reporting! I will try to limit the number of processes that can wait. Processes that have exceeded the limit will retry after waiting a certain time.

[edescourtis]

That's great thanks! Would you consider also having an option to make that time configurable and the waiter_pids limit configurable as well?

[edescourtis]

Let me know if you need help with anything.

[melpon]

@edescourtis Yes, I will make both configurable by config.exs or defmemo opts.

[edescourtis]

@melpon By the way, I really love how simple this library is to use. Good work!

[melpon]

Release memoize 1.3.0. You can use now.

In 1.3.0, number of waiter processes receiving message passing are limited. You can configure config.exs or defmemo opts. (prior defmemo)

With config.exs:

config :memoize,
  max_waiter: 100,
  waiter_sleep_ms: 1000

With defmemo opts:

defmemo foo(), max_waiter: 100, waiter_sleep_ms: 1000 do
  ...
end

[edescourtis]

Thanks so much for this. We will start testing it immediately and report any issues observed! Thank again!

[edescourtis]

We recently found a problem waiter_sleep_ms set to zero by default (see the new issue for details)