exclude and 'ResponseError: unknown command 'ZDIFFSTORE' '

[kuno]

I am on redis 2.4.8, This error message occurs when I try to use exclude method to filter some objects out.

Here is the fulll traceback:

In [18]: len(ActiveUser.objects.exclude(id__in=[1, 2, 4]).filter(id_in=[7, 8, 9]))

QuerySetError Traceback (most recent call last) /home/kuno/utopia/sogoke/sogoke/ in () ----> 1 len(ActiveUser.objects.exclude(id__in=[1, 2, 4]).filter(id_in=[7, 8, 9]))

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in len(self) 167 168 def len(self): -->169 return self.count() 170 171 # PRIVATE METHODS

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in count(self) 149 receive any data from the server. It construct the queries and count the 150 objects on the server side.''' -->151 self._buildquery() 152 return self.qset.count() 153

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in _buildquery(self) 178 if self.fargs: 179 self.simple = not self.filter_sets -->180 fargs = self.aggregate(self.fargs) 181 else: 182 fargs = None

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in aggregate(self, kwargs) 192 193 def aggregate(self, kwargs): -->194 return sorted(self._aggregate(kwargs), key = lambda x : x.name) 195 196 def _aggregate(self, kwargs):

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in _aggregate(self, kwargs) 206 if name not in fields: 207 raise QuerySetError('Could not filter on model "{0}".\ -->208 Field "{1}" does not exist.'.format(meta,name)) 209 field = fields[name] 210 value = (field.serialize(value),)

QuerySetError: Could not filter on model "streams.activeuser". Field "id_in" does not exist.

In [19]: len(ActiveUser.objects.exclude(idin=[1, 2, 4]).filter(idin=[7, 8, 9])) ERROR: An unexpected error occurred while tokenizing input The following traceback may be corrupted or invalid The error message is: ('EOF in multi-line statement', (77, 0))

---

ResponseError Traceback (most recent call last) /home/kuno/utopia/sogoke/sogoke/ in () ----> 1 len(ActiveUser.objects.exclude(idin=[1, 2, 4]).filter(idin=[7, 8, 9]))

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in len(self) 167 168 def len(self): -->169 return self.count() 170 171 # PRIVATE METHODS

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in count(self) 149 receive any data from the server. It construct the queries and count the 150 objects on the server side.''' -->151 self._buildquery() 152 return self.qset.count() 153

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/orm/query.pyc in _buildquery(self) 189 self.simple = False 190 self.qset = self._meta.cursor.Query(self,fargs,eargs, -->191 queries = self.queries) 192 193 def aggregate(self, kwargs):

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/backends/base.pyc in init(self, qs, fargs, eargs, timeout, queries) 35 code = self._sha.getvalue() 36 self._sha = code if not code else sha1(code).hexdigest() --->37 self.execute() 38 39 @property

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/backends/redisb.pyc in execute(self) 236 self.query_set = key 237 else: -->238 self.result = self.pipe.execute() 239 240 def order(self):

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/redis.pyc in execute(self, with_callbacks) 1091 conn = self.connection_pool.get_connection('MULTI', self.shard_hint) 1092 try: ->1093 return execute(conn, stack, with_callbacks) 1094 except ConnectionError: 1095 conn.disconnect()

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/redis.pyc in _execute_transaction(self, connection, commands, with_callbacks) 1050 parse_response = self.parse_response 1051 for i in range(len(commands)+1): ->1052 parseresponse(connection, '') 1053 # parse the EXEC.

1054 response = parseresponse(connection, '')

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/redis.pyc in parse_response(self, connection, command_name, _options) 233 def parse_response(self, connection, command_name, _options): 234 "Parses a response from the Redis server" -->235 response = connection.read_response() 236 if command_name in self.response_callbacks: 237 return self.response_callbacks[command_name](response, **options)

/home/kuno/utopia/sogoke/lib/python2.7/site-packages/stdnet/lib/connection.pyc in read_response(self) 181 response = self._parser.read_response() 182 if response.class == ResponseError: -->183 raise response 184 return response 185

[lsbardel]

Yes, that is because ZDIFFSTORE is not a standard redis command. I assume your ActiveUser has implicit ordering (Meta attribute ordering set to a model field). Check here for info http://lsbardel.github.com/python-stdnet/stdnetredis.html#sorted-sets

The 0.7 version will have a fallback using a lua script.

[kuno]

I assume your ActiveUser has implicit ordering (Meta attribute ordering set to a model field).

yes, it has.

It seems that to solved this issue I need either to use stdnet-redis branch or remove the implict ordering from my model. Do I understand you correctly?

[lsbardel]

With the current version yes, unfortunately. Once redis 2.6 is out, the new stdnet (version 0.7) will have a fall back for standard redis, via a lua script.

[kuno]

Hi Isbardel: I have more questions to ask, so I just re-opened this issue : ).

Since the 'exclude' method is pretty critical in my project, so I need to know more about it. Currently, as you knew, I can not use exclude method directly because I using official version of redis as backend of stdnet. Alternatively, I use the ifilter function of python itertool module to couple same task. (You can image how horrible it is in term of performance).

I am some confuses what the next should be do in order to solved this issue completely.

1, compatibility

As you mentioned early, either I need to use stdnet-redis or waiting for release of version 0.7. I just took a look on repository of stdnet-redis, the latest commit is about one month ago. Is this project still maintained?

Because redis 2.6 will come up with lua scripting, this bring some uncertainty for me. What the policy on the compatibility with official redis? Do you guy decide still maintain a separate redis branch or just using lua scripting to do all the compatibility works?

2, Upgrade If I decide to waiting for the release of 0.7, will this upgrade be seamless or I need do some extra works? I hope I can upgrade to 0.7 as soon as its release.

--best regards

--kuno

[lsbardel]

Version 0.7 will be fully compatible with the official release of redis 2.6 which is due in April. You won't need to worry about the stdnet-redis branch unless you are using timeseries. The zdiffstore command has been implemented in lua already

https://github.com/lsbardel/python-stdnet/blob/master/stdnet/lib/redis/lua/zdiffstore.lua

The upgrade to 0.7 is straightforward from the point of view of data. All you need to do is to reindex (save each instance of your models)

for o in mymodel.objects.query():
    o.save()

There are some changes in the API that might require some minor changes on users codebase. Look through the CHANGELOG for more information.

[kuno]

Thanks!

If I want to build stdnet-redis for emerging usage, could you offer some advice?

I saw there are two branch in stdnet-redis repository, stdnet-master and stdnet-2.4, which one should I use?

[lsbardel]

Same as building redis master. Download and run make. The usage is the same, but stdnet adds timeseries commands:

http://lsbardel.github.com/python-stdnet/stdnetredis.html#timeseries

the database file is also the same. In any case

• master->stdnet branch fully compatible, always.
• stdnet->master fully compatible unless you use timeseries commands.

[kuno]

Can not chained exclude? I just have a taste with exclude, it works fine. But it seems stdnet dose not support chained exclude/filter, for example

ActiveUser.objects.exclude(id__in=(1, 2, 3)).exclude(id__in=(4, 5, 6))

this queryset will ignore the first exclude only excute the second exclude. So you still will get the users with id 1, 2 ,3.

[lsbardel]

Indeed, this is a bug caused by the fact you are concatenating two queries with same keyword (id__in in this case). It is fixed in master. Thanks for reporting.

[kuno]

Thanks, Could you merge this fix into 0.6 branch and release a new version?

[kuno]

Another issue, It seems currently the filter/exclude method only support AND logic lookup. I mean every condition which passed to the filter/exclude method need to fulfilled simultaneously. For example:

ActiveUser.objects.filter(id=1, username='foo')

This queryset only return something when there is an user with id 1 AND username 'foo' exactly. What about the OR logic lookup? In the above example, how to make queryset return something when there is an user with id 1 OR an user with username 'foo'?

[lsbardel]

You are right, version 0.7 has the union method in a query which does that.

qs1 = mymodel.objects.filter(...).exclude(...)
qs2 = mymodel.objects.filter(...).exclude(...)
qs3 = qs1.union(qs2)

Version 0.7 is the real deal. It covers most situations and it is much much faster than 0.6.

[kuno]

exciting about 0.7, thanks. BTW: I am using 0.6 on pypy now, it seems works pretty well will pypy. Will 0.7 also works nicely with pypy?