Projection of fields in nested lists

[fkreis]

Dear spring-data-mongodb maintenance team, I am having trouble with the projection of fields in nested lists with spring-data-mongodb.

Given the following minimal example mongoDB:

db.nestedList.insertOne(
    {
      "flat": 1,
      "flatDescription": "one",
      "list": [
        {
          "element": 11,
          "description": "eleven"
        },
        {
          "element": 12,
          "description": "twelve"
        }
      ]
    }
 );
 db.nestedList.insertOne(
    {
      "flat": 2,
      "flatDescription": "two",
      "list": [
        {
          "element": 21,
          "description": "twentyone"
        },
        {
          "element": 22,
          "description": "twentytwo"
        }
      ]
    }
 );

Let's assume I only want to know about the fields "flat" and all "list.element". In mongoDB I can simply execute

db.nestedList
    .find({})
   .projection({ flat: 1, "list.element" : 1})

with the result

{
    "_id" : ObjectId("649a8045da17a6faee05e306"),
    "flat" : 1,
    "list" : [
        {
            "element" : 11
        },
        {
            "element" : 12
        }
    ]
},

{
    "_id" : ObjectId("649a80a9da17a6faee05e307"),
    "flat" : 2,
    "list" : [
        {
            "element" : 21
        },
        {
            "element" : 22
        }
    ]
}

This is exactly my expected and desired behavior!

Now my question is: How can I achieve this exact behavior with Spring Data (Java)?

The intuitive approach

ProjectionOperation projection = Aggregation.project("flat", "list.element");

generates

{ "flat" : "$flat", "element" : "$list.element"}

which produces

{
    "_id" : ObjectId("649a8045da17a6faee05e306"),
    "flat" : 1,
    "element" : [ 11, 12 ]
},
{
    "_id" : ObjectId("649a80a9da17a6faee05e307"),
    "flat" : 2,
    "element" : [ 21, 22 ]
}

which is obviously not what I need, as element is now a property in the root object and also is an array.

Slightly better

ProjectionOperation projection = Aggregation.project("flat");
projection = projection.and("list.element").as("list.element");

generates

{ "flat" : "$flat", "list.element" : "$list.element"}

which produces

{
    "_id" : ObjectId("649a8045da17a6faee05e306"),
    "list" : [
        {
            "element" : [ 11, 12 ]
        },
        {
            "element" : [ 11, 12 ]
        }
    ],
    "flat" : 1
},
{
    "_id" : ObjectId("649a80a9da17a6faee05e307"),
    "list" : [
        {
            "element" : [ 21, 22 ]
        },
        {
            "element" : [ 21, 22 ]
        }
    ],
    "flat" : 2
}

which is still wrong. At list the structure is correct now but list.element are still arrays.

I even tried

ProjectionOperation projection = Aggregation.project("flat");
projection = projection.and("list").nested(Fields.fields("list.element"));

generates

{ "_id" : "$_id", "list" : { "element" : "$list.element"}}

which produces

{
    "list" : [
        {
            "element" : [ 11, 12 ]
        },
        {
            "element" : [ 11, 12 ]
        }
    ],
    "_id" : ObjectId("649a8045da17a6faee05e306")
},
{
    "list" : [
        {
            "element" : [ 21, 22 ]
        },
        {
            "element" : [ 21, 22 ]
        }
    ],
    "_id" : ObjectId("649a80a9da17a6faee05e307")
}

which is still wrong as list.element are arrays.

Is this a bug or am I just using it wrong? In the latter case, can you please explain the proper usage? :) I am looking forward to you response!

Kind regards, fkreis

[christophstrobl]

Thanks @fkreis for reporting. Unfortunately there's no easy workaround that would result in what you're seeking. The issue is related to #3435. Let me see what we can do for you to support this scenario.

[christophstrobl]

There's an early branch for this. Meanwhile Aggregation#state could be an option.

Aggregation.stage("{ $project : { 'flat': 1, 'list.element': 1 } }")

[fkreis]

Hello @christophstrobl , thank you for looking into this! :blush:

I could imagine the following two approaches:

1. Change the behavior of ProjectionOperation projection = Aggregation.project("flat", "list.element"); : Instead of { "flat" : "$flat", "element" : "$list.element"} it should generate { "flat" : 1, "list.element" : 1} which for me would even be more intuitive.
2. Support something like CustomJsonAggregation, as suggested in #3435: Then I could enforce my desired behavior with Aggregation.newAggregation(CustomJsonAggregation("{ "flat" : 1, "list.element" : 1}"))). However, this suggestion was from May 31, 2021. Is there any plan of integrating this kind of feature?

I am looking forward to further discussions and exchange. Best, Fabian

[christophstrobl]

I'm not inclined to change the behaviour of a method that has been in service this way for 10+ years. What would be the benefit of writing Aggregation.newAggregation(CustomJsonAggregation("{ "flat" : 1, "list.element" : 1}")) over Aggregation.newAggregation(Aggregation.stage("{ $project : { 'flat': 1, 'list.element': 1 } }"))?

[fkreis]

I'm not inclined to change the behavior of a method that has been in service this way for 10+ years.

Absolutely reasonable! If other options are available we want to avoid breaking changes of course! :+1:

What would be the benefit of writing [...]

There is no benefit, I just was not aware of the Aggregation.stage feature. This is exactly what I meant with approach 2 - Thank you! :blush: Please note that https://github.com/spring-projects/spring-data-mongodb/issues/4428#issuecomment-1623630166 was a reply to https://github.com/spring-projects/spring-data-mongodb/issues/4428#issuecomment-1623306705. We both posted an update almost simultaneously.

[fkreis]

I just want to confirm that Aggregation.stage("{ $project : { 'flat': 1, 'list.element': 1 } }") as suggested in https://github.com/spring-projects/spring-data-mongodb/issues/4428#issuecomment-1623628410 works as expected in my real world project.

Thanks again for that workaround!