Memory error in aggregations

[marcelinhov2]

Hello, I'm trying to make an aggregation but the api is returning this error: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in.

I already tried to pass this allowDiskUse property but I got no success. Someone already solve it?

Thanks!

[mkjsix]

@marcelinhov2 unfortunately I don't think we handle that case at all.

@ujibang I guess we need to decide how to pass the allowDiskUse parameter dynamically, in terms of the present API design for aggregations. The Mongodb's Javascript API uses the db.collection.aggregate(pipeline, options) while I checked the code at org.restheart.handlers.aggregation.GetAggregationHandler and that parameter should be passed explicitly, by the Java fluent API as AggregateIterable.allowDiskUse(true)

Present code fragment:

                case AGGREGATION_PIPELINE:
                    AggregateIterable<BsonDocument> agrOutput;
                    AggregationPipeline pipeline = (AggregationPipeline) query;
                    try {
                        agrOutput = getDatabase()
                                .getCollection(
                                        context.getDBName(),
                                        context.getCollectionName())
                                .aggregate(
                                        pipeline
                                                .getResolvedStagesAsList(
                                                        context.getAggreationVars()))
                                .maxTime(Bootstrapper.getConfiguration().getAggregationTimeLimit(), TimeUnit.MILLISECONDS);

See: http://mongodb.github.io/mongo-java-driver/3.7/javadoc/com/mongodb/client/AggregateIterable.html#allowDiskUse-java.lang.Boolean-

[ujibang]

Hi,

I just added allowDiskUse option to the aggregation definition in commit d84b84a8b970559f141dab040730503315d8d650

Example:

{
    "type":"pipeline",
    "uri": "/test",
    "allowDiskUse": true,
    "stages": [
        "<stage_1>",
        "<stage_2>",
        ...
    ]
}

@marcelinhov2 can you please test it in you use case?

[marcelinhov2]

It's updated at your docker image already?

[marcelinhov2]

@ujibang I redeployed my restheart API using your original docker image but it didn't take any effect. I'm still getting the error.

[marcelinhov2]

I deployed again using the image softinstigate/restheart:3.4.0-SNAPSHOT and, apparently, it worked.

[mkjsix]

The image is built automatically. Did you pull the very latest image? I suggest to even delete the docker image locally and then pull it again, to be sure it's not using the locally cached image.

[mkjsix]

Yes, softinstigate/restheart:3.4.0-SNAPSHOT is the latest build

[ujibang]

I added allowDiskUse reference to documentation page https://restheart.org/learn/aggregations/

[marcelinhov2]

Great job guys, thank you so much