SHDICT enhancement proposal

[Thomas12]

Hi,

the current shdict implementation is nice and fast but lacks some important features.

Here a proposal for a shm-dict storage enhancement which would bring openresty's shdict implementation directly into the top-range of shm-caching due to its versatility, APIs or other custom enhancements should not be needed anymore:

sd:set(123456, 'bla'); sd:setchild(123456, 'age', 15); sd:setchild(123456, 'name', 'John'); sd:set_alias_key(123456, 'Employee Level 4'); sd:commit(); --to do all accumulated tasks in one blocked transaction (simply means all above happens while your shdict-mutex is continuously closed for others)

ret=sd:get(123456); pkey=ret['pkey']; --123456 akey=ret['akey']; --Employee Level 4 val=ret['val']; --bla age=ret['age']; --15 name=ret['name']; --John

also works by alias key:

ret2=sd:get('Employee Level 4'); pkey=ret2['pkey']; --123456 akey=ret2['akey']; --Employee Level 4 val=ret2['val']; --bla age=ret2['age']; --15 name=ret2['name']; --John

The indices pkey, akey and val are "reserved".

sd:del(123456); --automatically deletes all childs, and the alias key "Employee Level 4"

or

sd:del('Employee Level 4'); --automatically deletes all childs, and main key 123456

param='>='; ret=sd:get(123455, param);

param can be '<=', '>=', '=' (default '=') to find the exact key or the next bigger one or the next smaller one until the best matching key is finally found, if no matching key exists, nil is returned.

Easily iterate over all keys, even over key-series with gaps in it as the next-best is found:

next_key=0; while ret=sd:get(next_key, '>=') do if(ret==nil or next_key>100) then break; end; ngx.say('Found key: '..ret['pkey']..'\n'); next_key=ret['pkey']+1; end;

What do you think?

Thomas

[agentzh]

These features are interesting! They remind me of those Redis commands ;)

We're going to open the C API of ngx_lua's shared_dict so that people (or ourselves) can write custom Nginx C modules to extend it or even replace the implementation with something completely new :)

I'd really hope to keep the default shared_dict implementation in the ngx_lua core as simple as possible :) And adding the capability of extending or overriding the default engine will not stop us from adding new features freely :)

Thanks! -agentzh

[rik1083]

This is a very usefule feature indeed

I would really like something to persist the DICT across restarts e.g. dump contents to file periodically, and load from file on new start up

Or even simpler, just a way to list all the key:val without having to know al the key's externally, in order to script my own backups

foreach (entry e in DICT) { print(e.Key .. ":" .. e.Value) }