[leo_gateway][leo_mcerl] Hidden Memory Cache Capacity/Object Size Limit

[windkit]

Description

There is implicit lower bound of memory cache capacity in leo_gateway, you need at least 8*1024*1024 bytes (8 MB) for each cache worker.

On the other hand, there is a implicit upper limit for memory cached object at 8*1024*1024 bytes (8 MB)

Users are not aware about the limitation and fail to meet the requirements actually shuts down the memory cache.

Implicit Lower Bound for Memory Cache Capacity

cache.cache_ram_capacity = [NO LESS THAN] (cache.cache_workers) * 8 * 1024 * 1024

We can test with leo_mcerl unit test. At https://github.com/leo-project/leo_mcerl/blob/develop/test/test_leo_mcerl.erl#L40 Setting the capacitiy to 8 * 1024 * 1024 - 1 causes test case to fail.

Effect in GW

Memory Cache does not work, leo_cache_api:put/2 always fails.

$ ./bin/leo_gateway remote_console
Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:16:16] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V6.4  (abort with ^G)
(G0@192.168.100.35)1> leo_cache_api:put(<<"test">>,<<"abc">>).
{error,out_of_memory}

(p.s. oom is always returned when there is error in leo_mcerl/c_src/leo_mcerl_nif.c)

Underlying Cause

At https://github.com/leo-project/leo_mcerl/blob/develop/c_src/leo_mcerl_nif.c#L49

  lcu_cache_init(cache, auto_eject_on, max_size, 2, 256, 1024 * 1024 * 8);

Setting the max to other value would adjust the minimum requirement accordingly.

Implicit Upper Bound for Memory Cached Object

cache.cache_disc_threshold_len = [NO MORE THAN] 8 * 1024 * 1024

Effect in GW

Object larger than 8 MB but smaller than cache.cache_disc_threshold_len would not be cached (Not large enough for Disk Cache, Too large for Memory Cache)

$ ./bin/leo_gateway remote_console
Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:16:16] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V6.4  (abort with ^G)
(G0@192.168.100.35)1> Bin = crypto:rand_bytes(8388609).
<<93,118,55,51,95,208,163,141,125,110,133,171,55,68,252,
  227,97,170,42,202,0,194,52,9,176,39,248,40,143,...>>
(G0@192.168.100.35)2> leo_cache_api:put(<<"test">>,Bin).
{error,out_of_memory}

(G0@192.168.100.35)3> Bin2 = crypto:rand_bytes(8000000).
<<184,228,159,116,87,140,207,44,81,69,102,213,234,126,172,
  6,38,228,142,182,180,1,54,107,13,141,253,48,199,...>>
(G0@192.168.100.35)4> leo_cache_api:put(<<"test">>,Bin2).
ok

Underlying Cause

At https://github.com/leo-project/leo_mcerl/blob/develop/c_src/leo_mcerl_nif.c#L49

  lcu_cache_init(cache, auto_eject_on, max_size, 2, 256, 1024 * 1024 * 8);

Summary

Inside leo_mcerl/c_src/leo_mcerl_nif.c, there is a hard-coded limit 8 * 1024 * 1024 (8 MB)

cache.cache_ram_capacity / cache.cache_workers  
>= 8 MB 
>= cache.cache_disc_threshold_len

[windkit]

Action

1. Update the documentation about the limitation
2. Add a parameter check during Gateway boot up

[yosukehara]

Fixed

In case of invalid ram-cache configuration

%% log/app/error
[E] gateway_0@127.0.0.1 2016-12-22 12:11:46.320087 +0900    1482376306  leo_gateway_app:after_process_0/1   226 
    [{cause,{{invalid_configuration,"(8 * 1024 * 1024) * ${cache.cache_workers} >= ${cache.cache_ram_capacity}"},
    [{leo_gateway_app,get_options,0,[{file,"src/leo_gateway_app.erl"},{line,576}]},
     {leo_gateway_app,after_process_1,2,[{file,"src/leo_gateway_app.erl"},{line,255}]},
     {leo_gateway_app,after_process_0,1,[{file,"src/leo_gateway_app.erl"},{line,221}]},
     {application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}]

In case of invalid disc-cache configuration

%% log/app/error
[E] gateway_0@127.0.0.1 2016-12-22 13:39:17.881906 +0900    1482381557  leo_gateway_app:after_process_0/1   225 
    [{cause,{{invalid_configuration,"(8 * 1024 * 1024) * ${cache.cache_workers} >= ${cache.cache_disc_capacity}"},
    [{leo_gateway_app,get_options,0,[{file,"src/leo_gateway_app.erl"},{line,582}]},
     {leo_gateway_app,after_process_1,2,[{file,"src/leo_gateway_app.erl"},{line,254}]},
     {leo_gateway_app,after_process_0,1,[{file,"src/leo_gateway_app.erl"},{line,221}]},
     {application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}]