Pagination not working - Pass options.removeUndefinedValues=true to remove undefined values

[rastislavkopal]

Unable to use pagination using next tokens

I am trying to implement pagination using GSI. First find query works fine without providing next. When I add next param, i am getting error Pass options.removeUndefinedValues=true to remove undefined values from map/array/set.

I have following schema

export const InstallationSavesSchema = {
  format: "onetable:1.1.0",
  version: "0.0.1",
  indexes: {
    primary: { hash: "PK" },
    GSI1: { hash: "G1PK", sort: "G1SK" },
    GSI2: { hash: "G2PK", sort: "G2SK" },
  },
  params: {
    typeField: "_type",
    timestamps: true,
    isoDates: true,
    createdField: "createdAt",
    updatedField: "updatedAt",
  },
  models: {
    InstallationSave: {
      PK: { type: String, hidden: true, value: "installationUuid#${installationUuid}#itemUuid#${itemUuid}" },
      installationUuid: { type: String, required: true },
      itemUuid: { type: String, required: true },
      typename: {
        type: String,
        enum: Object.values(InstallationSaveTypes),
        required: true,
      },
      placeUuid: { type: String, required: true },
      createdAt: { type: Date },
      updatedAt: { type: Date },
      G1PK: {
        type: String,
        hidden: true,
        value: "installationUuid#${installationUuid}#placeUuid#${placeUuid}",
      },
      G1SK: {
        type: String,
        hidden: true,
        value: "typename#${typename}",
      },
      G2PK: {
        type: String,
        hidden: true,
        value: "installationUuid#${installationUuid}",
      },
      G2SK: {
        type: String,
        hidden: true,
        value: "createdAt#${createdAt}",
      },
    },
  } as const,
};

When I use find method on this schema without next, it works fine

const saves0 = await this.modelInstallationSave.find(
      {
        G1PK: `installationUuid#${installationUuid}#placeUuid#${placeUuid}`,
      },
      {
        limit: pageSize,
        index: "GSI1",
      }
    );

// output of saves0.next
   // next =  "next": {
   //     "G1SK": "typename#LANDMARK",
    //    "PK": "installationUuid#123#itemUuid#SAO_PAULO_CASA_DAS_ROSAS",
    //    "G1PK": "installationUuid#123#placeUuid#9ed982b6-5ef0-43fe-b178-27e44acae5fd"
    //}

just right after this query, i exec it again, with the next provided from this result as

  const saves = await this.modelInstallationSave.find(
      {
        G1PK: `installationUuid#${installationUuid}#placeUuid#${placeUuid}`,
      },
      {
        limit: pageSize,
        next: saves0.next,
        index: "GSI1",
      }
    );

And for the second request i am getting error Pass options.removeUndefinedValues=true to remove undefined values from map/array/set.

Onetable version: 2.6.2 Node: v20.11.1

[mobsense]

I don't think value templates in the PK work like you think they work.

In your find request, you should be providing the ingredients (installationUUid, placeUUid) and not the PK itself.

[rastislavkopal]

    const saves0 = await this.modelInstallationSave.find(
      {
        installationUuid,
        placeUuid,
      },
      {
        limit: pageSize,
        index: "GSI1",
      }
    );

    ConsoleDebug({ msg: "saves0", saves0, next: saves0.next });

    const saves = await this.modelInstallationSave.find(
      {
        installationUuid,
        placeUuid,
      },
      {
        limit: pageSize,
        next: saves0.next,
        index: "GSI1",
      }
    );

    ConsoleDebug({ msg: "saves", saves });

Right, I must read more about the template literals. But even when I use it just like this. I am getting same output.

Output of the first:

{
    "msg": "saves0",
    "saves0": [
        {
            "installationUuid": "123",
            "itemUuid": "SAO_PAULO_CASA_DAS_ROSAS",
            "typename": "LANDMARK",
            "placeUuid": "9ed982b6-5ef0-43fe-b178-27e44acae5fd",
            "createdAt": "2024-07-03T14:07:17.302Z",
            "updatedAt": "2024-07-03T14:07:53.728Z"
        },
        {
            "installationUuid": "123",
            "itemUuid": "SAO_PAULO_CITTA_DI_MAROSTICA_PARK",
            "typename": "LANDMARK",
            "placeUuid": "9ed982b6-5ef0-43fe-b178-27e44acae5fd",
            "createdAt": "2024-07-03T14:12:15.370Z",
            "updatedAt": "2024-07-03T14:12:15.370Z"
        }
    ],
    "next": {
        "G1SK": "typename#LANDMARK",
        "PK": "installationUuid#123#itemUuid#SAO_PAULO_CITTA_DI_MAROSTICA_PARK",
        "G1PK": "installationUuid#123#placeUuid#9ed982b6-5ef0-43fe-b178-27e44acae5fd"
    }
}

And for the second request I am still getting error

"errorMessage": "Pass options.removeUndefinedValues=true to remove undefined values from map/array/set.",

[mobsense]

Can you provide a debug.ts so we can replicate?

[mobsense]

Closing for now. Please feel free to reopen if you can supply a reproduction.

[ebisbe]

@mobsense I had the same issue. After digging a lot I have found that the sort property is mandatory with the last change from https://github.com/aws/aws-sdk-js-v3/issues/2158

The proposed solution by AWS is not working in this repo:

const dynamoClient = DynamoDBDocumentClient.from(new DynamoDBClient({
    apiVersion: "2012-08-10",
    region: process.env.AWS_REGION
}), {
    marshallOptions: {
        removeUndefinedValues: true
    }
});

because the https://github.com/sensedeep/dynamodb-onetable/blob/2de9a37c1d6b8c8ad466c1bf714c6f376040de66/src/Table.js#L1070 and https://github.com/sensedeep/dynamodb-onetable/blob/2de9a37c1d6b8c8ad466c1bf714c6f376040de66/src/Table.js#L1096 are getting the value from an incorrect place.

Here https://github.com/sensedeep/dynamodb-onetable/blob/2de9a37c1d6b8c8ad466c1bf714c6f376040de66/src/Expression.js#L600 if the defined index has not sort it's creating an object with {undefined: undefined, other properties } and because AWS changed how the marshall/unmarshall works this error is triggered.

[mobsense]

Is this the solution you propose:

                    let {hash, sort} = this.index
                    let start = {[hash]: cursor[hash]}
                    if (sort && cursor[sort]) {
                        start[sort] = cursor[sort]
                    }

[ebisbe]

That could be a solution to not add undefined: undefined but anyway lines 1070 and 1096 are wrong and not doing anything.