Uniqueness within ListFields

[mmellison]

Feature Request

Description

Uniqueness within ListFields would allow MongoEngine to assert uniqueness of lists containing fields and embedded documents, within the same document.

Background

Currently MongoDB does not enforce unique indices within the same document. For example:

class EmbDoc(EmbeddedDocument):
    x = StringField(required=True, unique=True)

class Doc(Document):
    l = ListField(EmbeddedDocumentField(EmbDoc))

d = Doc(l=[EmbDoc(x='t'), EmbDoc(x='t')]).save()  # No Exception

The saved document will contain duplicate entries for x, despite it being marked as unique. This is to be expected and is the normal behavior of MongoDB as of today.

Looking at http://docs.mongodb.org/manual/core/index-unique/ we can see the following:

The unique constraint applies to separate documents in the collection. That is, the unique index prevents separate documents from having the same value for the indexed key, but the index does not prevent a document from having multiple elements or embedded documents in an indexed array from having the same value.

More information about the issue can be found at http://joegornick.com/2012/10/25/mongodb-unique-indexes-on-single-embedded-documents/

MongoDB has had an open issue to add support for this since 2010. https://jira.mongodb.org/browse/SERVER-1068

Proposed Solution

I propose that until MongoDB addresses the above linked issue, if they ever do, uniqueness with ListFields is asserted via MongoEngine implementation.

If MongoDB is to ever address the issue, the internal implementation for MongoEngine can be switched to use Unique 'Doc' Indices in the same manner that unique fields are handled today via unique collection indices.

Solution Conceptualization 1

Add a doc_unique Boolean parameter to Fields .

class EmbDoc(EmbeddedDocument):
    x = StringField(required=True, unique=True, doc_unique=True)

class Doc(Document):
    l = ListField(EmbeddedDocumentField(EmbDoc))
   s = ListField(StringField(), doc_unique=True)

d = Doc(l=[EmbDoc(x='t'), EmbDoc(x='t')]).save()  # Now raises an exception.
d = Doc(s=['Conflict', 'Conflict']).save() # Now raises an exception.

Pros:

• Simple to understand.
• No syntax or identifier to parse.
• Contained within the Embedded Document (the parent doesn't need to know the doc_unique field).

Cons:

• Parameter would serve no purpose unless the field is inside of a ListField, which can lead to user confusion.

---

Solution Conceptualization 2

Add a doc_unique String parameter to ListField. The Id would be a string containing a '$' wildcard character used to represent an index in the list.

class EmbDoc(EmbeddedDocument):
    x = StringField(required=True, unique=True)

class Doc(Document):
    l = ListField(EmbeddedDocumentField(EmbDoc), doc_unique='$.x')
    s = ListField(StringField(), doc_unique='$')

d = Doc(l=[EmbDoc(x='t'), EmbDoc(x='t')]).save()  # Now raises an exception.
d = Doc(s=['Conflict', 'Conflict']).save() # Now raises an exception.

Pros:

• Only need to add a new parameter to a single field type.
• Can only be used when it is allowed to be used.

Cons:

• Parsing an identifier.
• Human readability and user usability.
• Parent must be made aware of any Embedded Document fields that need to be document unique.

[mmellison]

Any feedback or comments are appreciated. In many of my past projects, I just added a validation method to check for duplicates but IMO it would make things a lot cleaner if it was handled internally by MongoEngine.

[MRigal]

Without being able to choose a better solution, I think this feature would be great. I'm also doing the same with own validate method, it's a pity mongo doesn't support it directly, but it would be great to have this out-of-the-box with mongoengine

[mmellison]

Hopefully we see some support in MongoDB for this problem soon. I see https://jira.mongodb.org/browse/SERVER-1068 is now marked as in "Planning Bucket A". But who knows what that means :)

[MRigal]

This means actually nothing... it won't come out before ages...

I would else vote for Concept1: less cons :-p

[mmellison]

Yes -- it could take years before we see that feature implemented in MongoDB, but such is the nature of OSS. I will be revisiting this feature in the near future. There are some larger changes I want to propose first that would make feature like that a bit easier to implement.

[nitinbhojwani]

Is something going on for this?

[Adityashaw]

Any feedback or comments are appreciated. In many of my past projects, I just added a validation method to check for duplicates but IMO it would make things a lot cleaner if it was handled internally by MongoEngine.

Does anything exist for forcing uniqueness within EmbeddedDocumentListField or I have to manually check it? Edit : add_to_set does exactly what I needed. Thanks!