redis driver

[czechboy0]

It would be nice to also create a Redis driver for Fluent. I use Redis for most of my services as both the cache and the db.

[tanner0101]

Who else in @qutheory/contributors would like to see this? If it's a priority, I can work on it after SQLite and MongoDB are solid.

[p4t5h3]

I doubt that it makes sense. Redis is a key value store (imagine a database table restricted to two columns) and drastically different from relational databases like MySQL or NoSQL like MongoDB. In addition Redis typically holds data is in-memory with persistence being only an option. Further a restriction is that the data cannot be larger than host memory, a limitation you do not have with other database systems.

It is an excellent storage for a (yet to come) Vapor cache feature but not as a base for an ORM.

[…] People that are wrapping ORM’s around Redis are missing the point.

The killer feature of Redis is that it allows you to perform atomic operations on individual data structures, like counters, lists, and sets. The atomic part is HUGE. Using an ORM wrapper that retrieves a "record", updates values, then sends those values back, removes the atomicity, cutting the nuts off the major advantage of Redis. Just use MySQL, k?

Quote from https://github.com/nateware/redis-objects. Further: http://redis.io/topics/faq

[czechboy0]

I agree with @alomvar that if we wrap Redis in Fluent, we might not be using all 100% of capabilities of Redis, but isn't this true of all other database systems? I think that inherently if you create a lowest-common-denominator abstractions across many solutions, you always lose some capability to take the fullest advantage of any particular solution. But that's not the goal here, as far as I understand. I think the goal is to allow for users to write their logic around Fluent, without having to care which particular db is backing it.

Redis can store many things in memory, but it has settings for both memory storage limits and file disk saving frequency, so I don't think that's an issue.

I love using Redis for the raw speed of getting information out of it, which is much faster than most other solutions (it's mostly a smart cache, after all). All of these features can be taken advantage of even when wrapped by Fluent. Plus, as the only method that needs to be implemented by a Redis Fluent driver is

public func execute<T : Model>(query: Query<T>) throws -> [[String : Value]]

we can even use Lua scripts to send a complex, multi-step script over to Redis, which executes it and returns results. This is extremely performant and very common on high-traffic servers. And if we have a fast language (Swift), why not also have a fast datastore easily accessible from it.

I appreciate your points, but I think you might be focusing too much on the small negative things (which you can say about any database system), but I believe there is a huge set of advantages of having a fast cache/db available with the Fluent interface.

(And I don't see why it'd hurt to have the driver, as it's absolutely optional for anyone to use it, and there are people who would love to. Or am I missing some big issue that could be caused by having this driver?)

[Prince2k3]

Excellent response @czechboy0. Though I am more curious if Fluent should support more than one Driver. I've seen Backends use both redis and SQL/NOSQL db together. If a dev wants to do that for there setup how can they use Fluent to do this. Also wondering if I should create a different issue for that topic? But since folks are talking about Redis I brought it up

[czechboy0]

@Prince2k3 Yeah that's a great point, but I think it should be in a new issue, as that'd be a great feature (think cascading queries). This is purely about adding a Redis driver.

[p4t5h3]

I think that inherently if you create a lowest-common-denominator abstractions across many solutions, you always lose some capability to take the fullest advantage of any particular solution.

By pulling key-value storage in as a backend for the ORM the lowest common denominator gets lowered significantly. As I understand it would event prevent essential features of a good ORM like relationships and queries (how to restrict a result set by an attribute? In other words: how to SELECT * FROM table WHERE column > 100 with a key-value store? How to implemented let user = Query<User>().filter("id", notIn: [1, 2, 3]).filter("age", .GreaterThan, 21).first from Fluent readme?).

I think the goal is to allow for users to write their logic around Fluent, without having to care which particular db is backing it.

I agree, that is one another essential piece of a database abstraction layer.

Redis can store many things in memory, but it has settings for both memory storage limits and file disk saving frequency, so I don't think that's an issue.

Transaction safety is not guaranteed by design. The advantage if Redis is how it works in-memory. The synchronization is based on snapshots. You can write to disk at every query, but it basically annihilates the performance advantage and you just have a "dumb" database.

we can even use Lua scripts to send a complex, multi-step script over to Redis, which executes it and returns results.

If you do it within the driver it is okay. Otherwise:

I think the goal is to allow for users to write their logic around Fluent, without having to care which particular db is backing it.

It would be counterproductive to the goal of abstraction.

but I believe there is a huge set of advantages of having a fast cache/db available with the Fluent interface. And I don't see why it'd hurt to have the driver, as it's absolutely optional for anyone to use it, and there are people who would love to.

As far as I read there is only the performance advantage from the point of view of a database abstraction. And that can also be approached with SQLite in memory. It is easier to integrate due to its more affine nature to this use case compared to key value stores.

If a dev wants to do that for there setup how can they use Fluent to do this.

Redis could be integrated as a built in cache for Fluent. However that would be in competition to a more generic, separate from Fluent cache feature and thus questionable code duplication. I see only these two options:

• Use a database designed for persistence and transaction safety as storage in combination with redis for caching abstracted on ORM level (Fluent) (eventually redundant and not modular on a higher level)
• Use a database designed for persistence and transaction safety as storage in combination with redis for caching abstracted on application level (Vapor) with requires custom wiring

Using redis for persistent storage is like running your office server on an tablet. You can do it and there sure are people who want to do it – but is it worth the hassle? I would recommend waiting for an actual demand for this database driver.

[czechboy0]

I would recommend waiting for an actual demand for this database driver.

Yes! Here it is, I created this issue for this exact reason! 😊

I appreciate you going so deep into why SQL is better than Redis under Fluent, that might as well be true, but why does that prevent us from having a Redis driver? This is like arguing which programming language is the best one 😆 (in a group of 10 engineers you'll have 11 opinions).

✅ There is demand for it ✅ It is doable, and if done well with Lua scripts can be extremely performant ✅ It's optional for anyone to use it

You are absolutely correct that SQL might be the better solution in some cases, but there are many other cases in which Redis would have a more appropriate performance curve, like in many of my server apps. I want that in-memory layer that gives me great performance.

As far as I read there is only the performance advantage

That's a big advantage in my book, probably the biggest reason why I got into Redis in the first place. I very much care about performance, but everyone can have different priorities. And everyone should be allowed to use what fits them best, shouldn't they? We all have preferred tools, because we like them or we're used to them. Either way, if people want to use Fluent with Redis, why do you feel like you have to discourage them? 😊

[tanner0101]

Are there any Swift + Redis connectors out there that Fluent could use?

I'm considering having an in between for cache layers and Fluent. Something like CacheDriver that could better support the high performance and uniquely key:value needs of caches. Redis could then conform to this cache driver.

From there, Fluent could also conform to the cache driver allowing projects to use MySQL, SQLite, MongoDB, etc for their caches as well.

[Prince2k3]

@tannernelson @czechboy0 has https://github.com/czechboy0/Redbird that he created

[p4t5h3]

[…] why SQL is better than Redis […]

That is not my point. It is about how a key-value store is not as suitable as an ORM storage as everything else (including other NoSQL databases) because design and intended use differs. As I wrote earlier, relationships and querying, natural features of ORMs in my experience, are (kind of) possible to implement but not reasonably.

Otherwise I am wondering why not skip the outsourcing to redis and keep the database entities in memory. I would expect that to happen anyway since the application itself persists over requests unlike (for example) PHP applications.

@tannernelson The official redis site has a section about Swift clients.

[czechboy0]

@tannernelson That sounds like a great approach! That way some of us could still use Redis as a regular db, but others could seamlessly use it as a cache for their db. Count me in! And yeah, I wrote Redbird, which features a streaming parser, so it should be hella fast and low on memory usage. And it's all pure Swift, no C libs under the hood. It's listed as the first Swift client on the official Redis client list.

[tanner0101]

💯 Will definitely use Redbird.

app.cache is on the Vapor roadmap for version 10 or 11 (working on 9 right now). So I will probably be bugging you with questions in a week or so. :P

[czechboy0]

Happy to help, I'm glad Redbird will be useful to more people, it's already running on a few of my Swift (yes Vapor) servers in production. 👍

[tanner0101]

We have this now in Vapor https://github.com/vapor/redis-provider

[czechboy0]

@tannernelson Very happy to see this get in. I can't wait for Vapor 0.17.0 with Cache support, will be a great release 🎉

[seivan]

Usually, look at Rails for good API. I also need to use Redis new Geo-stuff (instead of Mongo)