Feature request: Namespace support

[mjrusso]

Proposal:

r = redis.Redis(namespace='foobar')

Behind-the-scenes, this would prefix every key with foobar.

For example, if I issued the following command:

r.set('abc', 'xyz')

It would be equivalent to the following:

redis-cli set foobar:abc xyz

In Ruby, the redis-namespace class does the same thing.

However, I think that it would be preferable if this were a core feature of the Python client.

[andymccurdy]

This seems like a reasonable suggestion. I'll add it in the next version. Remember that the namespace string will add extra bytes to each key. It might be better in some cases to namespace with the db=X option, since it has no additional memory overhead.

[kmerenkov]

What is the use case in real life (development or production)?

[mjrusso]

This allows the user to run several "applications" on top of a single Redis install in a manner that is more flexible than using different databases and different Redis processes.

This feature becomes increasingly useful as the number of utilities built on top of Redis increases, because namespace support allows multiple apps to be run against a single Redis install without fear of keys overlapping.

For example, Resque ( http://github.com/defunkt/resque ), uses namespacing to enable users to run multiple Resque instances in a straightforward manner. (More details are in the Namespaces section of the README.) The feature also lets this same Redis instance be used for a different purpose, in addition to Resque.

It's most useful in a development context though. Right now I'm using Redis for over a dozen different purposes and I need to do this constant juggling act on my main dev machine (and, in general, be very careful that keys don't collide).

[kmerenkov]

Okay so I came up with an ugly but obvious solution. http://github.com/kmerenkov/redis-py/tree/namespaces

I use a decorator and decorate nearly all commands.

Pros: obvious Cons: uncool :-)

[andymccurdy]

@kmerenkov: Check out the consistent_hashing branch in my redis-py repo. I needed a way to detect which arguments in a command were keys just like you do, but did it a bit differently, similar to the way response callbacks work. I was envisioning using that for namespacing as well, barring a more elegant solution.

[dottedmag]

I feel I probably too late to suggest it, but such namespacing is easily done as a separate wrapper. Crude attempt is available at http://gist.github.com/616851 - it needs more special-casing for functions which don't follow (key, *args) signature, but it works pretty well for me.

[nikegp]

Hey Andy, any updates on this feature? It's really useful for development when you have many developers working on the same instance. Especially if you have many redis instances in your topology

Thanks

[binarymatt]

I started down the road of doing a wrapper that adds namespacing on top of redis-py. Here are the beginnings of what I was thinking: https://gist.github.com/2190617. It's not complete, but if there is still interest in adding ns to redis-py, i'd love to get feedback and possibly switch this over to something like a pull request.

[zmsmith]

I started playing around with an implementation here:

https://github.com/zmsmith/predis/blob/master/predis/client.py

It's based on StrictRedis using consistently named parameters, but it could definitely be changed to be more configurable. Also, it doesn't alter any calls that return keys right now, but it's a start.

[grimnight]

Any progress on this? It would be nice to have namespaces.

[softwaredoug]

I have a Python project, subredis, that implements this nicely. A Subredis is a nearly full StrictRedis implementation/wrapper that uses a namespace. Its definitely a useful pattern. Some thoughts on limitations:

• Lua scripting becomes challenging. How do you replace references to keys in Lua scripts with a namespaced key? In subredis I don't allow this.
• How much do you sandbox the namespace? Should the namespaced redis be able to do admin-like commands/config like bgsave, etc? In subredis, I don't allow this.

[paicha]

2010-2015

[naktinis]

@andymccurdy You've mentioned you could "add it in the next version". Is it still an option? I believe it would be a welcomed feature. Especially now that multiple databases are discouraged.

Edit: multiple databases might not be deprecated for non-cluster mode after all.

[Kudo]

+1

[joshowen]

+1

[thestick613]

+1?

[exit99]

@andymccurdy Here is a working version with tests. I can expand the testing to all commands if you like the direction. https://github.com/Kazanz/redis-py/commit/8994f2633e60294573cdbeac452802ed1002d24a

[andymccurdy]

@kazanz your solution only works for commands that deal with exactly one key. Commands like KEYS, SUNION/SINTER, SORT, and others that operate on 2 or more keys would not be covered. Also, any command that doesn't operate on a key, e.g. INFO, CONFIG etc. would also break with this change. It's one of the reasons why this issue is tricky and hasn't already been addressed.

[exit99]

@andymccurdy I see. If no one else is addressing the issue, I would be happy to. Let me know.

[exit99]

@andymccurdy Here is a working version that allows namespacing on every command along with tests for every command. Works in py2 & py3. Followed proposed syntax of @mjrusso. Pull request here.

[etcsayeghr]

6 years later, still no namespace?

[andymccurdy]

@etcsayeghr Pull requests are accepted. This is a pretty complicated issue that I personally don't have a need for.

You can see some of the complexities here: https://github.com/andymccurdy/redis-py/pull/710#issuecomment-174850384

[exit99]

@andymccurdy Finally have some more time to refactor based on your comments. Will post periodically for feedback.

[exit99]

Finished a working PR with namespace support for all cmd calls on:

• StrictRedis
• Redis
• PubSub
• Pipeline
• Lua Scripting (only for keys passed to the script)

Refactored based on @andymccurdy comments on my last PR.

New PR uses simple decorators to handle namespacing. Followed proposed syntax by @mjrusso.
Has modified readme with usage instructions. Be sure to look at readme as some previous code may need changing if you are applying a namespace.

Includes full test suit for python2 + python3.

Here is my response to the complexities mentioned.

Thanks

[exit99]

@andymccurdy Any update on this pull request? We use this in production at my company and would like to see it integrated so we don't have to maintain a separate installation.

[andymccurdy]

@Kazanz Hey, sorry I've been away visiting family and friends for the past 8 weeks. When I last looked at this, there were some commands that weren't implemented (SORT comes to mind) and some commands that looking at the code didn't seem like they'd work correctly (for example ZADD has multi=True, but that won't work if the score values are strings (the scores would get prefixed) and it also wouldn't work if someone passed name/score pairs as **kwargs).

After discovering a few of these types of issues I went off into the weeds and made a separate implementation. I made execute_command take a parameter called keys_at. The value can be one of 3 things:

• an int that is the index of a single key in the args. commands like GET use keys_at=1, where the args are ('GET', 'my-key').
• a list of ints. same idea as above, but can specify multiple keys. great for commands like SORT where there's no defined pattern to where keys occur in the command string.
• a slice created with slice(). commands like MGET use keys_at=slice(1, None), and MSET uses keys_at=slice(1, None, 2).

keys_at tells us then specifically which args in a command string are keys so that they can be prefixed.

Although still incomplete (I haven't dealt with commands that return values like KEYS), I think it's much more straight forward and easier to understand what's happening. I'll make a branch and push my work.

[exit99]

@andymccurdy Having 8 weeks for family and friends must be nice :).

I like the keys_at parameter, would definitely be more readable.

Perhaps we could also have a keys_returned_at parameter for functions that could return keys?

When you get your branch up, I will go through and write defaults for each command and anything else you think needs to be addressed.

Happy to be moving forward on this.

[manusha1980]

@andymccurdy May I know the latest status of this please?

[exit99]

@andymccurdy Any chance your going to be pushing that branch soon?

Again, I am happy to go through and define the keys_at parameter for each command and anything else you think needs to be addressed.

If you havn't got a chance to write the code, I'm happy to write something along the lines of your previous post.

Let me know.

[AngusP]

Adding to this, it may make sense also supporting hash tags for those running a cluster of Redis servers https://redis.io/topics/cluster-spec#keys-hash-tags

[AngusP]

It may make sense having a keyname function (and inverse function) that is passed to redis, and called on all keys before they're sent off to Redis. This would also allow those that want to to check keys match their key name schema (e.g. all uppercase, no whitespace, properly delimited...)

If key namespaces were added as just a string passed to the class, we'd have to choose a delimiter that would then become the default for anyone using redis-py and derivatives. : seems commonly used but there are probably many others. As Redis has no real notion of namespaces and thus has no standard delimiter for key names, it would seem nice to avoid choosing one if possible.

[LouisKottmann]

It's 2k17 and that feature is still wanted :p

Any chance for it to see the light of day?

[exit99]

@AngusP I think this makes sense to use a set of functions and let the user dictate (and test) their own namespace logic. I think this would help backwards compatibility as well for users who would have multiple namespaces using the same instance in a project.

I am taking another stab at this, now that I have a little more time available.

[exit99]

@andymccurdy New pull request having namespace support. Feedback appreciated.

[exit99]

@andymccurdy Friendly reminder that PR #866 had been open for almost 2 months now with your recommended implementation. I will merge in updates from the past 2 months, but need confirmation on direction before exerting the additional effort. Thx.

[exit99]

Is it safe to assume that this feature is never going to be supported? A PR with the suggested implementation has been open for months, and is now falling behind master. If you are not intending to support please tell us now.

[andey]

2018 checking in. Please add namespace support.

[exit99]

@andey There was good back and forth between myself and the repo owner on the PR above. Its probably gone stale now, but at the time that PR added support. We've been using it for over a year now in production.

[kbni]

@andymccurdy Can you please weigh in on this? This would be a very useful feature to have!

[purplecity]

9 years later. still no namespace😁

[gencurrent]

Still need it. I write subclass everytime, hoping the feature will be released soon. Got dissapointed

[Jerry-Ma]

Year 2020. I'd love to see this feature get implemented.

[AngusP]

@Jerry-Ma please remember this is an open-source project and people are giving their time freely to work on it... in a sense this makes the contributors the customer, you need to sell them an idea if you want them to implement it.

That being said, I will probably PR a change that'll let you pass non-str/bytes objects as keys, and just call str on them, so namespaceing can be handled by external classes (if this is acceptable to @andymccurdy) like so (very quick illustrative example):

class NamespacedKey(object): 
    def __init__(self, namespace, key=None): 
        self._namespace = namespace 
        self.key = None
    def __str__(self): 
        return f'{self._namespace}:{self.key}'

# ...
k1 = NamespacedKey('namespace')
k1.key = 'keyname'

r = redis.client.Redis(...)
r.set(k1, 'test value') # currently raises an exception:
# DataError: Invalid input of type: 'NamespacedKey'. Convert to a byte, string or number first.

[andymccurdy]

@AngusP That seems like a pretty reasonable approach to me. I like that it has no performance penalty to folks who don't need namespaced keys.

If you're going the route of global key instances that are defined in some shared location and then imported/used in Redis commands, we'll also need a way to template the key such that runtime variables can be interpolated. For example, I might have a hash object per user identified by their user_id, e.g. user-{user_id}. The full key name, including namespace, might be my-namespace:user-123.

With that requirement, perhaps we just do something like:

# defined in some shared location...
namespace = Namespace('my-namespace:')
user_key = namespace.key('user-%(user_id)s')

# now we can use the key elsewhere...
r = redis.Redis()
user_id = ... # get the user_id from somewhere
user_data = r.hgetall(user_key.resolve(user_id=user_id))

This also has the added benefit of not needing special treatment in the redis-py Encoder. The key object returned by namespace.key() has a resolve() method that returns a string.

[Jerry-Ma]

@AngusP

I totally agree with your comments and I did end up implement the namespace on my own after I posted the comment.

I made use the key_pos idea proposed in this thread, and the key part of the code looks like the follows:

class _KeyDecorator(object):
    def __init__(self, prefix=None, suffix=None):
        self._prefix = prefix or ''
        self._suffix = suffix or ''

    def _decorate(self, key):
        return f"{self._prefix}{key}{self._suffix}"

    def decorate(self, *keys):
        return tuple(map(self._decorate, keys))

    def _resolve(self, key):
        return key.lstrip(self._prefix).rstrip(self._suffix)

    def resolve(self, *keys):
        return tuple(map(self._resolve, keys))

    def __call__(self, *args):
        return self.decorate(*args)

    def r(self, *args):
        return self.resolve(*args)

class RedisKeyDecorator(_KeyDecorator):
            pass

class RedisIPC(object):

    connection = StrictRedis.from_url(url, decode_responses=True)

    _dispatch_key_positons = {
            'get': ((0, ), None),
            'set': ((0, ), None),
            }

    def __init__(self, namespace=None):
        self._key_decor = RedisKeyDecorator(prefix=namespace)

    def __call__(self, func_name, *args, **kwargs):
        _key_pos, _key_return_pos = self._dispatch_key_positons[
                func_name]
        if isinstance(_key_pos, slice):
            _key_pos = range(*_key_pos.indices(len(args)))

        for i, a in enumerate(args):
            if i in _key_pos:
                args[i] = self._key_decor.decorate(a)
        result = getattr(
                self.connection, func_name)(*args, **kwargs)
        return result

[Jerry-Ma]

@AngusP

I like the idea of using str(obj) to make keys, but it seems cannot handle returned keys. If we were to support the key decorating, APIs like the follows would make sense to me:

class AbstractKeyDecorator(abc.Meta):

    @classmethod
    @abstractmethod
    def decorate(cls, key):
        """Return the decorated key."""
        return NotImplemented

    @classmethod
    @abstractmethod
    def resolve(cls, key):
        """The inverse of `decorate`."""
        return NotImplemented

class MyKeyDecor(AbstractKeyDecorator):
    # implement the interfaces
    pass

conn = StrictRedis(...)

# stateful
conn.set_key_decorator(MyKeyDecor)

value = conn.get("a")  # "a" get decorated
keys = conn.scan("*")  # all returned keys get resolved.

# or state less
value = conn.get("a", key_decorator=MyKeyDecor)
keys = conn.scan("*", key_decorator=MyKeyDecor)

[AngusP]

I suppose I'm maybe misinterpreting the use case -- there seem to be two slightly different ones

1. Encapsulation, so a particular client instance is can only touch keys with a certain prefix. This would include using the hash slot syntax in Redis
2. Convenience, reduce repeated code when using a complex naming scheme for keys

We could perhaps address both simply by adding optional hooks that can be used to transform keys on the way in (and sometimes on the way out of the client).

e.g.

def prefix(key):
    return f'my_namespace:{key}'

# Simple encapsulation example
r = Redis(..., key_hook=prefix)
r.get('test') # calls GET my_namespace:test

# More complex:
def prefix(key):
    return f'{{my_namespace}}:node_{os.uname().nodename}:{key}'

If you want your keys to all be instances of some class that does fancy things to key names, just pass key_hook=str and write your own resolution logic inside the class' __str__ or __repr__ methods.

In the convenience case, if a stateless hook is insufficient then it might just be easier to be explicit as in @andymccurdy's example user_data = r.hgetall(user_key.resolve(user_id=user_id)) and leave the logic outside of the client.

[andymccurdy]

I favor convenience for several reasons.

1. To me, the "encapsulation" approach leads to users believing that there is some security guarantee that they cannot possibly see keys outside of their configured namespace. But a simple r.execute_command('GET', 'other-namespace:secret-stuff') violates that. And while we'd never make such a guarantee, it's easy to see how users will (wrongly) assume that one exists.

2. Implementing this feature should have zero performance impact to users who don't use namespaces.

3. The "encapsulation" approach makes future command contributions more difficult. Committers and reviewers will have to remember to account for the namespace feature and ensure that new commands pass keys_at (or whatever key identification mechanism) correctly.

4. Similarly, it makes support for Redis module plugins more difficult for the same reason. For example, someone could make a "redis-py-search" project that mixes in command methods and response callbacks for all of the Redis Search module commands. Going the "encapsulation" route would force that plugin to also properly implement the same key identification logic.

5. The "convenience" option supports users who want to use r.execute_command() directly instead of the command methods.

6. The "convenience" option is simple and explicit.

7. The "convenience" option is simple for a user to override and implement their own logic if necessary.

If we go the "convenience" option, I'd be fine including those helpers within redis-py and documenting how to use them.

[github-actions[bot]]

This issue is marked stale. It will be closed in 30 days if it is not updated.

[abrookins]

I’d like to see this and I’d be happy to work on it. Sounds like @andymccurdy is ok with the convenience approach, which I could work on. Managing complex keys is an important part of most non-trivial projects, so we should give users some tooling to help with that and guidance on how to do it well.