Version 5

[ConstBur]

Hello Alec, it's me again. I'd just like to thank you again for making this driver and for incredible help in my last issue ( #21 ). Fred is awesome, the best among Redis drivers I've encountered. Never thought that someone could make something better pretty much single-handedly than the teams/communities behind other drivers, especially the "official" one (https://github.com/mitsuhiko/redis-rs).

As a thank you I'd really like to support this library, you've got Patreon or something like that?

Can't wait for the 5.0.0 release, hope it will be ready soon!

[xxated]

I'd like to support fred too!

[aembke]

Ah thank you both that is very kind of you to say. I don't have a patreon or anything, and as much as I appreciate the sentiment it's totally not necessary and I'm happy to continue working on this for free. I'm very fortunate in that my team uses this at Okta so I have access to financial and technical support from an awesome team there if needed.

Regarding version 5 - here's a mini state of the union on that:

For the most part it's finished, with the exception of some minor perf improvements that can likely wait until a later version down the road. The only real remaining work is finishing the streams (XADD, XREAD, etc) interface tests, which to be honest are a bit of a slog, but I'm not willing to give up the (nearly) 100% integration test coverage goal on the public interface. I keep thinking those will be done "soon", and then I discover another use case I want to test, and that has pushed this out further than I would have liked. That being said, I'm not really willing to compromise on those tests since the absolute last thing I want to do is wake up in the middle of the night to debug one of our big ElastiCache clusters.

In terms of new features or perf changes, here's what you can expect in version 5:

• The protocol parser got a rewrite so it can operate on data from the server without any moving or copying of the underlying bytes, which should be a big perf improvement for use cases that send or receive large payloads. If you use the bytes or bytes_utils crates in your app you can now leverage those types to avoid any moving or copying of data in your app as well.
• I had to rewrite the entire public interface with traits because deduplication via the Deref trait on the RedisClient struct was becoming unsustainable from a code maintenance standpoint. It's a lot easier to work with now, but since async functions are not yet supported in traits I had to switch back to using trait objects to hold return values, which is a very minor perf regression. In the end I think it's worth it though, and hopefully async functions in traits will be stable "soon".
• The streams interface is now implemented (XADD, XREAD, XREADGROUP, etc).
• Some perf improvements via the removal of some locks.
• RESP3 support (this was a big one).
• Moved a lot of the perf configuration flags from globals to client-specific config structs, and added a whole bunch of new perf config options for tuning backpressure, retry/backoff, etc.
• Made the cluster state management more reliable to handle some failure modes where lots of things go wrong with the cluster all at the same time.
• Fixed some bugs that can happen when redis keys are not UTF-8 strings.
• Added a serde-json feature to automatically convert to/from JSON values. You can pass a JSON object to HSET, HMSET, etc now, or have the library automatically convert an HGETALL response to an object, etc.
• Added a high level pubsub client that manages subscription state to automatically re-subscribe to channels when connections close, etc.
• Some docs improvements.
• Lots and lots of tests.

Aside from RESP3 support (which was my main goal with v5) a lot of these changes are under the hood to make this library more reliable. My main high level goal with this library is that if/when something does go wrong with the server that your app layer will just "stop" without losing any data while the server recovers, and once the server is back up the app layer will immediately start working again without any manual changes in your app layer, including any process restarts, etc. I think v5 makes some big steps towards achieving that, but there's still likely some work to be done to make that bulletproof.

If either of you are interested in contributing please feel free to let me know, I'm happy to take contributions or add maintainers. It's really just the streams tests holding up v5 at this point, but I know that's not very fun to work on so no worries if you'd rather not open that can of worms. Hopefully it'll be done in the next 2-3 weeks either way before this turns into the next Star Citizen.

[ConstBur]

Thanks for another detailed response, those are some spicy release notes!

Really appreciate that you're willing to keep developing this regardless of the situation, that kind of dedication is rare nowadays. And happy to know that you've got Okta backing this up :smiley:

Me and @xxated are currently doing a project where we use both Rust and Redis for the first time. So while we'd really like to contribute (and in the future we will), right now we can only really do so materially due to our substantial lack of knowledge in both, and I don't think we'll get to the necessary level in time.

So just to confirm, we can use the feat/resp3-support branch for now to start learning/using Streams?

[aembke]

Yeah feel free to start using that branch. Everything works so far, the only caveat is that I may still change some of the streams interface if I find it too difficult to work with in the tests. A lot of the ergonomics changes I've made in the last few months have come about from actually using the interface in tests, realizing that it's tough to use, and then changing it to make the tests easier. Since I still have to write tests for some of the streams commands I might change those.

[aembke]

I'm going to close this for now but I'll tag both of you in the v5 release when it's ready.