[Feature Request] Add a `STORE dstkey` option to all commands returning a result

[GoogleCodeExporter]

The idea arose while I was tackling the following problem:
let's say I have two ZSETs containing users sorted by age and height,

ZADD ages 20 "Joe"
ZADD ages 24 "John"
ZADD ages 22 "Jack"
ZADD ages 28 "Josh"

ZADD heights 180 "Joe"
ZADD heights 175 "Josh"
ZADD heights 165 "John"
ZADD heights 172 "Jack"

and I want to know who are the users who are between 20 and 25 yo, and who 
measure more than 170cm.
basically, what I need to do is compute the intersection of the ranges 
ages[20:25] and heights[170:]. at this point in time, the only way to do that 
is query ZRANGEBYSCORE ages 20 25, use the client to create a temporary set, 
store the result as a set, do the same thing for ZRANGEBYSCORE heights 170 
+inf, and then compute the set intersection.

at first, I thought I would propose a ZRANGEBYSCORESTORE command;
ZRANGEBYSCORESTORE tmp1 ages 20 25
ZRANGEBYSCORESTORE tmp2 heights 170 +inf
SINTER tmp1 tmp2

HOWEVER, this is a more general problem : given that a command returns a 
result, there *should* be a way to store that result in a key instead of 
sending it to the client.

Therefore, I propose the following extension:
CMD args [STORE dstkey]

if `CMD args` returns a (status code|error|integer|bulk) reply, store the 
result in `dstkey`
if `CMD args` returns a multi bulk reply, the type of the new value would 
depend on the context: S* args STORE dstkey would store a set, etc.
Incidentally, SINTERSTORE dstkey args would be equivalent to SINTER args STORE 
dstkey

A final note about how ZRANGEBYSCORE would work with store :
ZRANGEBYSCORE zset min max STORE dstkey 
-> store the result as a set in dstkey
ZRANGEBYSCORE zset min max withscores STORE dstkey 
-> store the result as a zset in dstkey

Why this would be a good thing :
- coherent api (if a command returns a result, then the result can be stored. 
not just for S*STORE)
- same syntax as for SORT ... STORE dstkey
- no trips between the client and the server to store temporary data

Original issue reported on code.google.com by adrien.friggeri on 14 Jul 2010 at 1:13

[GoogleCodeExporter]

Oooh... Interesting.

I really like this idea.

Original comment by jzaw...@gmail.com on 14 Jul 2010 at 1:22

[GoogleCodeExporter]

As gnrfan pointed out on irc, this could be extended to add a `STORENX dstkey` 
option which would do the same think if `dstkey` does not already existe.

Original comment by adrien.friggeri on 14 Jul 2010 at 1:23

[GoogleCodeExporter]

pietern also seems to find the proposal valuable but commented the 
implementation is not trivial so it needs a good deal of thought first. Let's 
see what antirez thinks about it :)

Original comment by gnr...@gmail.com on 14 Jul 2010 at 3:56

[GoogleCodeExporter]

Chatting with pietern a bit more his initial concern about this is, the 
conceptual gain of this must be carefully pondered so scripting features don't 
get added where they might not belong.

Original comment by gnr...@gmail.com on 14 Jul 2010 at 4:15

[GoogleCodeExporter]

as I stated on irc, I think that as long as we only consider read/store 
operations, we are not in the realm of scripting (no conditions, no loops, 
etc.).
imho, the conceptual gain is that there is no unnecessary communications 
between the client and the server when several complex operations are to be 
composed together (just store the result in temporary keys).

this being said, and after looking at the source, I understand it might be 
quite a headache to implement in this form (as it would require to modify all 
commands).

the alternative would be of this form : 
STORE dstkey CMD args

pop the STORE and dstkey, evaluate the command, and instead of sending the data 
back to the client, store it in dstkey. now, if I read correctly the source, 
the data is sent back unbuffered to the client…

a dirty solution would be to add a `storetype` flag to a request, which would 
indicate if the output has to be stored and if so, the type of data to store. 
then modify `addReply` and if `storetype` is not set to "do not store", 
bufferize the data at that level, according to the specified type. at the end 
of `processCommand`, if there is buffered data, store it. 
well, that's a very very rough idea, I'll investigate more if needed.

Original comment by adrien.friggeri on 14 Jul 2010 at 4:53

[GoogleCodeExporter]

Very interesting - could this also be used within MULTI/EXEC to chain commands 
on a single request?

Original comment by dr.marc....@gmail.com on 14 Jul 2010 at 10:15

[GoogleCodeExporter]

I need to think at this better. I don't like the general idea of ... meta 
commands, because it is somewhat a step towards an ad-hoc hill conceived 
scripting language. But anyway let's talk theoretically :) In order to 
implement such a thing without breaking the command semantics or without 
changing a lot of code something like this is needed instead:

STORE foo
LRANGE mylist 0 10

so it should be a command that sets the state to "let's store the next command 
output there".

Also in order to make this possible, internally Redis should be able to target 
the format of the object and not the protocol. This means that addreply() 
family functions should be check some client flag to understand if what to 
output is the protocol or a Redis object.

Also... I'm not sure that it is possible to simply output the same type as the 
input type... think at SORT for instance, and in general to all the things that 
will go from unordered to ordered outputs and vice versa. Or to sorted sets 
operations where the score is lost.

Btw such a feature will hardly be incorporated in such a way :(

This should be IMHO part of the scripting engine that is getting more and more 
clear in my mind (but is still far... clustering is the first priority after 
2.2), and will be the subject of the next weekly update.

Taking this issue open for a few weeks, it's an interesting issue and I would 
like to get some feedback.

Cheers,
Salvatore

Original comment by anti...@gmail.com on 30 Aug 2010 at 4:17

[GoogleCodeExporter]

This would be great to have. After first day with redis I was missing this kind 
of feature.

Would be great to run all the operations via redis without need to fetch data 
which is then used again to query redis. 

Original comment by hamsterr...@gmail.com on 25 Nov 2010 at 1:04

[GoogleCodeExporter]

I've run into this too, exactly adrien's example. I think being able to store a 
temporary is a huge win instead of transferring an arbitrary size list out, and 
then storing it again. 

Original comment by blackdo...@gmail.com on 29 Nov 2010 at 11:32

[GoogleCodeExporter]

Adding lua scripting to vim will help with these kind of issues, although this 
particular use case is big enough on it's own, I feel, to be added

Original comment by manchesterboy on 3 May 2011 at 9:36

[GoogleCodeExporter]

Was there ever a move on this? IMO it's the most obvious missing feature in 
Redis.

Original comment by robin.vi...@gmail.com on 12 Dec 2013 at 12:44

[GoogleCodeExporter]

Hi robin,
The issue list has moved to https://github.com/antirez/redis/issues
I dont know why this place is still online..

Original comment by mir...@gmail.com on 12 Dec 2013 at 1:04