Deepcopy - regex and Q querysets

[rozza]

Here's what I'm trying to do. Using a regex causes "TypeError: cannot deepcopy this pattern object"

import re MyDocument.objects(domain='amazon.com') if subject: filter(subject=re.compile('.*'))

Any suggestions? I realize there are some ways around this (i.e. build a dict and then do MyDocument.objects(**kwargs)), but I also need to include an OR expression:

from mongoengine.queryset import Q MyDocument.objects( (Q(subject=re.compile('text.'))| Q(subject=re.compile('other.')) & Q(domain='amazon.com')))

Which seems to require a deepcopy as it throws the same error.

[aclowes]

Ross,

Thanks for confirming this as a limitation. For the moment I'll use raw queries.

One idea would be to follow the Django example of using operators regex and iregex, which don't take pre-compiled expressions.

Alec

[rozza]

Good call - as you dont actually need to provide a regex to use the $regex operator.

[hassek]

Did this ever got implemented?

[chrislambert]

This is an issue for us. Don't see a way to work around it, even with raw queries. Was there a resolution?

[omidraha]

I have same issue, here is a sample code:

from mongoengine.connection import register_connection, DEFAULT_CONNECTION_NAME
from mongoengine import Document, StringField
from mongoengine.queryset import Q
import re

class Sample(Document):
    name = StringField(max_length=200, required=True)
    family = StringField(max_length=200, required=True)

    def __unicode__(self):
        return '{}.{}'.format(self.name, self.family)

print(mongoengine.VERSION)
# (0, 8, 7)

register_connection(DEFAULT_CONNECTION_NAME, 'test')

Sample.drop_collection()

s = Sample(name='omid', family='raha')
s.save()

print(Sample.objects.all())
# [<Sample: omid.raha>]

r = re.compile('omid', 1)
q = Q()
q.query["name"] = r

print(Sample.objects.filter(q))
# [<Sample: omid.raha>]

print(Sample.objects.filter(q, family='raha'))
# raise `TypeError: cannot deepcopy this pattern object`

[v-b-r]

Is this issue fixed ? If not, Experts - please fix it !! :)

[mindojo-victor]

Here is how to use raw query:

mongoengine.Q(__raw__={'email': {'$regex': email_pattern}})

But it doesn't work with negation: https://jira.mongodb.org/browse/SERVER-13779

[oblivionguns]

Hi guys,

I ran into the same issue and unfortunately I can't use {'$regex': <pattern>} because as @mindojo-victor said, you cannot negate the expression (required by my use case).

After multiple workaround attempts I found a pretty decent solution thanks to the bson.regex.Regex class ans its Regex.from_native method:

So in @omidraha's example, you'd use it like:

from bson.regex import Regex

r = Regex.from_native(re.compile('omid', 1))
q = Q()
q.query["name"] = r

Hope this fixes your issue as it did for me :-)

[mindojo-victor]

@oblivionguns Does your approach work with regex negation? Because it's MongoDB server limitation.

[oblivionguns]

Yes it does because it's equivalent to (in mongo shell):

{"name": {$not: /exemple/}}

or in Python to:

query["name"] = Regex.from_native(re.compile('omid', 1))
# or
query["name"] = re.compile('omid', 1)  # which should work with pymongo directy

This allows to negate a regex (which you can't with the $regex operator as you pointed out).

[mindojo-victor]

Wow! Looks like it's working:

In [1]: import mongoengine as me

In [2]: import bson.regex

In [3]: from *** import User

In [4]: User.objects.count()
Out[4]: 2548

In [5]: User.objects(me.Q(email={'$not': bson.regex.Regex('example.com$', 'i')})).count()
Out[5]: 2533

Thank you!

[mindojo-victor]

Strange that this doesn't work:

In [8]: User.objects(me.Q(email__ne=bson.regex.Regex('example.com$', 'i'))).count()

...
OperationFailure: command SON([('count', u'user'), ('fields', None), ('query', {'email': {'$ne': Regex('example.com$', 2)}})]) on namespace brain2.$cmd failed: Can't have regex as arg to $ne.

[oblivionguns]

Yeah that is a MongoDB limitation. Negating regex must be done with $not it seems.

Moreover, that would have a rather stranger meaning : "not equal to " while the stored field is not a regex. Did you try with email__eq ? (Wouldn't make more sense in my opinion :-p)