Support for embedded documents (aka subdocuments)

[aknoerig]

First of all thanks for this fine and very useful library. Given that it's surprisingly difficult to make FastAPI/pydantic work seamlessly with MongoDb, this effort is very much welcome!

Is there any plan to support embedded documents?

MongoDb supports two basic patterns for modelling slightly more complex data structures: relations and embedded documents. Relations are already well-supported in beanie via the Link type, but I could not find any mentioning of how to make use of embedded documents.

[roman-right]

Hey @aknoerig , Thank you :)

You can do smth like this:

class Inner(BaseModel):
    num: int

class Sample(Document):
    inner_lst: List[Inner]

Will this work for you? Or you are looking for some additional methods for this? If so, pls provide some details about your use case. Thank you.

[aknoerig]

Yeah, what you mention is great for creating nested/embedded fields. An embedded document is more than that, as it would have its own unique id (locally unique, but could also be globally unique). This offers easy access to these sub-elements, provides checks for uniqueness of id, etc.

So in your example it would have to be:

class Inner(Document):
    num: int

class Sample(Document):
    inner_lst: List[Inner]

For reference, see how mongoose treats "subdocuments".

[roman-right]

In the first documentation example, it has no unique ids:

{
  "_id": "joe",
  "name": "Joe Bookreader",
  "addresses": [
    {
      "street": "123 Fake Street",
      "city": "Faketon",
      "state": "MA",
      "zip": "12345"
    },
    {
      "street": "1 Some Other Street",
      "city": "Boston",
      "state": "MA",
      "zip": "12345"
    }
  ]
}

But yes, it can have unique ids. For this you need to add a factory for the id field like next:

class Inner(Document):
    id: UUID = Field(default_factory=uuid4)
    num: int

class Sample(Document):
    inner_lst: List[Inner]

Unfortunately, Beanie doesn't support automatically added unique indexes for the embedded docs, but it can be set up on the parent doc using the path to the embedded id field (in the inner Collection class).

Using this you'll be able to save the Inner document to the separated collection and the whole copy will be stored to the Sample doc on inserts/updates of the Sample doc.

Action-based events will not work on the saving as the internal doc, as the save/insert and etc method will not be called separately for these docs.

I'll take a look at the mongoose implementation - probably I'll implement some patterns from there in Beanie too. Thank you :)

[aknoerig]

That's interesting, thanks.

I guess what I would like to be able to do is an easy way to do CRUD on these embedded docs. Something along these lines:

sample.inner_lst.append(Inner(1))
inner = sample.inner_lst.get(inner_id)
sample.inner_lst.set(inner_id, inner_updates)
sample.inner_lst.delete(inner_id)
sample.save()

[cheradenine]

I'm looking for this as well. Porting some code from Ruby + Mongoid where they support embedded documents, you can declare a Document class and give it an attribute like 'embedded_in' and reference another Document. When these are written to the DB they get ids, actually called "_id" just like every other document. I have created an 'EmbeddedDocument' class inheriting from BaseModel and gave it an id field like this:

class EmbeddedDocument(BaseModel):
    id: Optional[PydanticObjectId] = Field(default=PydanticObjectId(), alias="_id")

But it seems Beanie is filtering out the field on serialization because it is called "_id". If I remove the alias, the id field gets written to the DB. I maybe be able to cope with this but the challenge I have now is that I'm going to python code writing documents and ruby code reading them so I'm worried about compatibility. I can see why Beanie would strip out _id on a Document, but it looks like it is doing this as well on a sub-field of a root document model.

[athulnanda123]

@roman-right Odmanic (another python odm) have this feature. https://art049.github.io/odmantic/modeling/, after migrating from odmantic to beanie, this is one of the biggest issue I am facing.

[xeor]

I'm trying to find an odm for my next project and this project looks better for me than odmantic.. The only thing missing is better many to many and foreign key alike structures like this :/

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been stalled for 14 days with no activity.

[aknoerig]

Please consider reopening this issue. As a workaround, I'm currently trying to work directly with arrays of subobjects, but the support for that is also rather weak currently. Several of the standard MongoDB queries for accessing and manipulating array elements are obscured by the high-level beanie API.

[roman-right]

Hi @aknoerig , Could you please share the queries you do to me understand the use case better? Maybe I can add a special Generic for this case to tell Beanie to handle such relations.

Like:

class Door(BaseModel):
    height: int
    width: int

class House(Document):
    door: Embedded[Door]

And the same for lists : List[Embedded[...]]

But I want to understand the use case a bit better to get which queries should be covered

[penggin]

I'm not this issue's owner, but I want this feature for better structure. Here is my case:

class User(BaseModel):
    uid: int
    nickname: str

class UserGroup(Document):
    gid: int
    users: List[User]

Without this, I have to update UserGroup (which includes every user in the user group) to update one user's nickname, and it's looks so bad.

[slingshotvfx]

+1 for this feature. I also need to be able to update a nested model by it's nested ID.

That said, I think you might be able to accomplish this with array_filters:

await UserGroup.find_one(UserGroup.id == group_id).update(
            {"$set": {"users.$[u].nickname": "new name"}},
            array_filters=[{"u.uid": user_to_update.uid}],
        )

But it would be great to have a nicer way to do that natively with Beanie

[slingshotvfx]

Actually this is even simpler with mongo's positional $ operator

await UserGroup.find_one({UserGroup.id: group_id, "users.uid": user_to_update.uid}).update(
            Set({"users.$.nickname": "new name"}),
        )