Any way to do key match on Cache?

[eminarcissus]

I'm now trying to use Concache to replace redis in the project, it works pretty well. But one feature I'm missing here is redis.keys pattern. Basically since all keys in redis is binary based, for erlang I'm planning to use a tuple key like {name,age,index} = value and put into ConCache.

If I extend it using the ets match/select query I think this can be done for some pattern matches with the key, but I don't know the impact if will have to the performance, or if any drawbacks it might have in regards to TTL mechanism.

Any suggestions will be really helpful, thx for your efforts to create this project :)

[sasa1977]

If you can do it with :ets.match or :ets.select than that's probably the best option unless the table is very large. These operations are going to be O(n), but at least they're performed in the C code, which will filter all the things you need, and return only the data you require. This is good, because data is always copied from the ETS table, so you don't want to copy the data you don't need. See here for more info.

Match patterns won't work for every case. If you need to e.g. do a regex match, or some more complicated conditional, then your best option is to iterate through keys, do conditional logic on each key, and fetch the value if needed. You can do this with :ets.first and :ets.next. Doing so will prevent from allocating the memory for all keys.

Iterating with first/next might also be beneficial if the table is very large. In such cases :ets.match will take longer which might skew the scheduler (because a single match/select is performed atomically, i.e. without preemption).

Either way, you'll need to invoke ets operations directly. You can get the table handler via ConCache.ets. If you'll be accessing the table in a tight loop, I suggest you fetch the handler before the loop.

Keep in mind that ets operations don't have any impact on ttl, which will not be renewed. Also, since ttl expiry is done from another process, it's possible some entries will be deleted immediately after you've fetched them. Finally, if you're iterating key by key, you should prepare for the situation where you fetch a key, but the value is not there anymore, because it's just been deleted.

That's simply a consequence of the concurrent nature of the structure. ConCache doesn't guarantee strong consistency. It's meant to be used for caches, where data serves as an "optimization hint". If it's there you use it to avoid some computation, but if it's not you should always be able to recompute it from the real source of truth (such as database).

Hope this helps, and let me know if you have more questions.

[eminarcissus]

Thanks for the tip, that's really really helpful, for current stage I think pattern match is quite enough, I will poke with the lib on console first to do some test first, and see if possibly also add a key matching function as well.

For another question is cache backup and restore, and that's being answered in another thread as well, but I didn't delve much into the code yet so I don't know how to restore it from a dets table. if I do have a copy of the current cache's ets table(by using :ets.to_dets),when restore it back from restart, should I directly call :dets.to_ets to restore it back from persistence, will it also bring back TTL information as well?

[sasa1977]

If you need to match on partial key values (such as "get me all entries where age > x"), then it should be simple. Let me know if you get stuck, and I'll try to cook some example.

TTL table is not persisted. ConCache happened because I needed it for my own project, so it has only what I needed at the time :-) When it comes to restoring the data, I just iterated all persisted entries and put them to cache. This is basically resetting of ttl of all values to the global default.

If you have the need for TTL persistence, let me know, and I'll think about providing the feature, though I'm not sure how simple/complicated would the implementation be.

[eminarcissus]

I've tested with :ets.select and match and it worked really well. I also used Ex2ms.fun to help creating a MatchSpec func like :ets.select t,(fun do {{"tim",age,_},y} = z when age > 10 -> z end) ,and now everything works like a charm.

For TTL persistence if a fix can be done it would be appreciate, but if that does provided much more complexity maybe just let it go, restore the data with default ttl is good enough :-), currently it already saved me from quite a lot of headache with redis drivers XD

[sasa1977]

Just as an fyi, you could do the same without using Ex2ms. In your module you need to add:

@compile {:parse_transform, :ms_transform}

Then from any function of that module you can do

:ets.fun2ms(fn({{"tim",age,_},y} = z) when age > 10 -> z end)

[eminarcissus]

That's really helpful, thx for the tips :) 2016年2月13日(土) 0:24 Saša Jurić notifications@github.com:

Just as an fyi, you could do the same without using Ex2ms. In your module you need to add:

@compile {:parse_transform, :ms_transform}

Then from any function of that module you can do

:ets.fun2ms(fn({{"tim",age,_},y} = z) when age > 10 -> z end)

— Reply to this email directly or view it on GitHub https://github.com/sasa1977/con_cache/issues/19#issuecomment-183396315.