StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.87k stars 1.51k forks source link

Missing Commands in StackExchange.Redis #2055

Open slorello89 opened 2 years ago

slorello89 commented 2 years ago

Hi @NickCraver @mgravell , as we discussed a few weeks ago (with @chayim and @gkorland) I went through the library and compiled a list of what I believe are the missing commands of StackExchange.Redis.

Methodology

I basically diffed RedisCommand.cs with the main Redis Project's commands.json, which resulted in ~180 commands, I went through and purged most of the commands that felt like they were admin/config (except ACL) to preserve the more interactive commands, this left ~70. Then I went through the library to see if the sub-commands that came up (a lot of stuff from XGROUP XINFO and a few others) were already covered by a combination of the command Enum + a literal to filter those out as they are already supported. After that, and collapsing the many sub-commands of FUNCTION and ACL, we're left with 47. I think what follows is mostly correct (in that the thing is ACTUALLY missing, and that it's something that you would probably want to add to the library if it were feasible). I marked the blocking commands, and then applied a bit of my own reason to determine if I thought the command would be addable without a major overhaul of the library. I also added the Redis Version the command was added in (about 40% of the missing commands are 7.0, I don't know what your policy is on adding Release candidate commands, I'm guessing you wouldn't want to :) )

As it stands, there are about 21 commands which are in stable versions of Redis, interactive, and shouldn't require any major architectural changes, call them low-hanging fruit. Seems like there are a few commands that were either just missed, or there's some story behind why they never made it in (BITFIELD, some OBJECT subcommand, SCRIPT KILL, LOLWUT (ok maybe a bit silly, but can always use a bit of levity)). And then I'm guessing there hasn't been a command refresh since 6.0

This is meant as a starting point, for my part, I'm happy to start opening PRs for any commands you'd approve of adding. I understand the reviews can be burdensome and do not want to overwhelm you, so just let me know what you want to be added and I'm happy to get started.

Full List

All the missing commands, including 7.0 commands and things that are going to be a bigger headache to add.

Command Name PRs Interface Proposed names Notes
ACL (+sub-commands) 6 (TBD) Skip: Broader discussion
BITFIELD #2107 3.2 IDatabase StringBitFieldSet/StringBitFieldGet/StringBitFieldIncrement/StringBitField This API might be tricky, the command is variadic, so an array of sub-commands may make sense, or the three sub-commands split out, or both could work. Also, might want a clever return-type.
BITFIELD_RO #2107 6.2 IDatabase see notes Consider intelligently interrogating the commands executed to see if they are RO.
BLMOVE 6.2 IDatabase Skip: Blocking
BLMPOP 7 IDatabase Skip: Blocking
BZMPOP 7 IDatabase Skip: Blocking
BZPOPMAX 5 IDatabase Skip: Blocking
BZPOPMIN 5 IDatabase Skip: Blocking
~COPY~ #2064 6.2 IDatabase KeyCopy
EVALSHA_RO 7 IDatabase ScriptEvaluateReadOnly
EVAL_RO 7 IDatabase ScriptEvaluateReadOnly
~EXPIRETIME~ #2083 7 IDatabase KeyExpireTime
FCALL 7 IDatabase Skip: Pending functions design
FCALL_RO 7 IDatabase Skip: Pending functions design
FUNCTION (+sub-commands) 7 IDatabase Skip: Pending functions design
~GEOSEARCH~ #2089 6.2 IDatabase GeoSearch
~GEOSEARCHSTORE~ #2089 6.2 IDatabase GeoSearchAndStore
~GETEX~ #1743 IDatabase StringGetSetExpiry
~HRANDFIELD~ #2090 6.2 IDatabase HashRandomField
~LCS~ #2104 7 IDatabase LongestCommonSubsequence
~LMOVE~ #2065 6.2 IDatabase ListMove
~LMPOP~ #2094 7 IDatabase Overload ListLeftPop and ListRightPop
~LPOS~ #2080 6 IDatabase ListPosition/ListPositions
~OBJECT ENCODING~ #2088 2.2 IDatabase KeyEncoding
~OBJECT FREQ~ #2105 4 IDatabase KeyFrequency This can only be used if maxmemory-policy is set to an LFU policy, might be worth some command-map manipulation to interrogate this policy.
~OBJECT REFCOUNT~ #2087 2.2 IDatabase KeyRefCount
~PEXPIRETIME~ #2083 7 IDatabase KeyExpireTime Overload that specifies to get the expire-time in milliseconds
PUBSUB SHARDCHANNELS 7 IServer SubscriptionShardChannels
PUBSUB SHARDNUMSUB 7 IServer SubscriptionShardSubscriberCount
SCRIPT KILL 2.6 IServer ScriptKill
~SINTERCARD~ #2078 7 IDatabase SetIntersectionLength This could also be SetCombineLength, I worry though because there's no corresponding union/diff commands yet that this can lead to some confusion.
~SMISMEMBER~ #2077 6.2 IDatabase Overload SetContains
~SORT_RO~ #2111 7 IDatabase SortReadOnly Another option is to just change the Sort method to use the SORT_RO, but I fear that's a break (and there would need to be some other logic to determine if SORT_RO is available on the Redis Server
SPUBLISH 7 (TBD) Skip: Pending sharded pub-sub design
SSUBSCRIBE 7 (TBD) Skip: Pending sharded pub-sub design
SUNSUBSCRIBE 7 (TBD) Skip: Pending sharded pub-sub design
WAIT 3 (TBD) Skip: Blocking
~XAUTOCLAIM~ #2095 6.2 IDatabase StreamAutoClaim
~ZDIFF~ #2075 6.2 IDatabase SortedSetCombine
~ZDIFFSTORE~ #2075 6.2 IDatabase SortedSetCombine
~ZINTER~ #2075 6.2 IDatabase SortedSetCombine
~ZINTERCARD~ #2075 7 IDatabase SortedSetIntersectionLength Same comment as with SINTERLENGTH
~ZMPOP~ #2094 7 IDatabase Overload SortedSetPop
~ZMSCORE~ #2082 6.2 IDatabase Overload SortedSetScore
~ZRANDMEMBER~ #2076 6.2 IDatabase SortedSetRandomMember
~ZUNION~ #2075 6.2 IDatabase SortedSetCombine
NickCraver commented 2 years ago

Hey Steve! This is awesome. My thoughts are:

  1. Yes, I'd like to add these and would welcome PRs.
  2. I'd like to ideally get #2041 in first because it gives us a lot of checks in the compiler for free and in applying it, I recognize a lot of missteps we allowed in result processors e.g. with streams that would have been caught.
  3. Working cross-fork is messy, I'd rather add you as a contributor here (main branch is protected, so no one can mess it up without a PR/check already).

I don't think ground rules for these are an issue, your first PR was fantastic. I propose that we pick a branch naming scheme (e.g. feature/<command>), and ideally lay all of these our as method names since we have the list (to ensure they make sense as a whole). In the past, we added as Redis did and it's a bit organic and not everything matches - IMO it'd be a good exercise to lay these out as the return type and method name(s). This helps because when the return types match for 2 very similar commands, sometimes it's better as an argument. If the return types are different (e.g. long for count vs. actual results), it's obvious they'll never be the same method, and both help inform the naming.

I'm happy if we want to setup a call and go through these or whatever you think works to get those down. IMO the naming is worth an hour to go through ahead of time so we're not figuring it out and conflicts/etc. each PR - much less time overall :)

Thoughts?

NickCraver commented 2 years ago

Updating #1743 to get GETEX off the list here :)

slorello89 commented 2 years ago

Hey @NickCraver , totally agree, let's come to an agreement on what the API will look like before we get underway. Can set up a call Next week or early (Mon/Tues) the week after.

slorello89 commented 2 years ago

@NickCraver, Striking GETEX given #1743 has been merged.

NickCraver commented 2 years ago

@slorello89 hmm, at random I went to https://redis.io/commands/zunion/ - the results...aren't what I expect there in the console. General feedback though: the command interface on the new site is nice looking and has good info, but is far less usable. I'm having to go back and re-search when trying to find the command(s) I want, something that was much better in the previous interface. I'm not sure where to put that feedback, but wanted to pass it along!

slorello89 commented 2 years ago

Hey @NickCraver, yeah, seeing some of the same with some other commands.

I believe redis-clinterwebz is the home of the app now. @itamarhaber would be the definitive voice on that. Will look at it tomorrow.

slorello89 commented 2 years ago

Hey @NickCraver RE laying out what the API will look like, I put together the following table (have a copy in sheets as well) as basically a jumping-off point to kick off this discussion. Happy to go back and forth asynchronously, happy to get on a call too (though I'm going to be out-of-pocket Wed-Fri this week.

EDIT

Moved updated table up to the full table in the first comment.

ttingen commented 2 years ago

@slorello89 I'm happy to help with this list if you're interested. @NickCraver sorry for all the nulls you had to fix in streams for #2041, my fault... :/

Avital-Fine commented 2 years ago

@slorello89 Did you mean SortedSetCombineAndStore for ZDIFFSTORE command?

NickCraver commented 2 years ago

@slorello89 I'm happy to help with this list if you're interested. @NickCraver sorry for all the nulls you had to fix in streams for #2041, my fault... :/

No worries at all! That was me being irked at me for not catching a few things in PR, we'll keep reving this thing better and better and I'm very, very appreciative of the help :) Now we've inboarded the tooling to help us all better as we add more commands and processors so yay. I'm super happy to find some badness in various places with the tooling lit up...give me decent confidence in the wins.

NickCraver commented 2 years ago

@slorello89 I went through and updated the list with PR pointers - recommend we collapse your later comment with naming into the original because it's either method names or blocking/punt so that can be 1 column so we can more easily update things to track here - thoughts? Happy to collapse that if agreeable.

slorello89 commented 2 years ago

Hey @NickCraver, I collapsed everything as requested, and noted the commands that need more consideration:

  1. ACLs
  2. Functions - the Redis 7 function feature
  3. Shard pub/sub - the Redis 7 Shard Pub-sub feature.

I think probably any command that I've named (with the exception of PUBSUB SHARDCHANNELS and PUBSUB SHARDSUBSCRIBERS) is probably fair game at this point @NickCraver?

I'd also note that we should be careful to separate anything going out in Redis 7, There's a case to be made for having those PRs ready to go (those features are increasingly set), but all the same I don't think any tests written against them are worth anything until Redis 7 is GA.

@Avital-Fine, @ttingen or anyone else working this, I'd say just shout out here what commands you want to work on before getting underway, that way there's no duplication of effort.

I'm going to start on the following today:

PS I met @ttingen at Codestock last week - was nice catching up, he was super excited to hear about the work happening with the library :)

Avital-Fine commented 2 years ago

I will start now with EXPIRETIME and PEXPIRETIME commands. Also, I know that there are new features to EXPIRE, PEXPIRE, EXPIREAT, PEXPIREAT so I'll check it out :)

Avital-Fine commented 2 years ago

Btw, after we finish with the commands we need to check for missing features. For example, I notice that SortedSetAdd (ZADD) doesn't support GT and LT options which were added in 6.2.0

NickCraver commented 2 years ago

Heads up: it occurred to me we're not actually testing 7.0-rc bits because of the Docker container version. I remedied that in #2079 which should be followed by a merge of main into any PRs with 7.0 features so we're properly exercising those paths :)

ttingen commented 2 years ago

I will start with the following: XAUTOCLAIM ZRANDMEMBER

NickCraver commented 2 years ago

@ttingen ZRANDMEMBER is up in #2076 ;)

ttingen commented 2 years ago

@ttingen ZRANDMEMBER is up in #2076 ;)

Oops, let's make that XAUTOCLAIM and ZMSCORE.

Avital-Fine commented 2 years ago

Working on OBJECT REFCOUNT and OBJECT ENCODING

slorello89 commented 2 years ago

I'll pick up:

Avital-Fine commented 2 years ago

Starting working on LCS

ttingen commented 2 years ago

I have XAUTOCLAIM working, just need to write more tests. Should be able to get it out for review tonight.

tyler-boyd commented 2 years ago

re: sharded pub/sub, is it as simple as changing PUBLISH commands to SPUBLISH etc? Or is there some drawback to the sharded version that requires more thought?

mgravell commented 2 years ago

IIRC we already have equivalent routing code, so it shouldn't be a huge change - however, there are non-trivial validations needed here, to ensure that we route correctly for both initial and reconnect scenarios, and to check the routing isn't impacted for non-routed subscriptions.

caoyang1024 commented 1 year ago

Hi, can I check the plan on commands (SPUBLISH, SSUBSCRIBE, SUNSUBSCRIBE) for sharded pub-sub.