Optimizing Performance: implementing Pagination before lookup and projections in large collections

[albertorizzi]

I have a collection containing 15,000 documents. Due to numerous lookups, pagination becomes significantly large. In the provided code snippet, I follow the documentation's guidance by performing aggregation before pagination.

However, this approach does not lead to any performance improvement because aggregation is executed prior to pagination.

Is it possible to first implement pagination and then apply lookup and projection to this paginated data?

const getPaginatedArcheologicalFindFromDb = async (
  optionPagination: PaginateOptions
): Promise<AggregatePaginateResult<ArcheologicalFindDocument>> => {
  try {
    let aggregate = ArcheologicalFindModel.aggregate(
      [
        ...cratedUsSectorSiteDigPipelineStage,
        ...archeologicalFindClassesPipelineStage,
        ...archeologicalFindTypesPipelineStage,
        ...archeologicalFindFunctionalShapePipelineStage,
        ...archeologicalFindShapePipelineStage,
        ...archeologicalFindProjectionTablePipelineStage,
      ],
      { allowDiskUse: true }
    );

    let archeologicalFind = (await ArcheologicalFindModel.aggregatePaginate(
      aggregate,
      optionPagination
    )) as AggregatePaginateResult<ArcheologicalFindDocument>;

    if (!archeologicalFind || archeologicalFind.docs.length === 0) {
      throw new AppError(
        StatusCodes.NOT_FOUND,
        "Nessun reperto archeologico trovato"
      );
    }

    return archeologicalFind;
  } catch (error) {
    throw new AppError(error.httpCode, error.message);
  }
};

[aravindnc]

See if new version resolves this issue.