clarkie / dynogels

DynamoDB data mapper for node.js. Originally forked from https://github.com/ryanfitz/vogels
Other
490 stars 110 forks source link

BatchGetItems multiple requests doesn't work #78

Closed RodH257 closed 7 years ago

RodH257 commented 7 years ago

This feature on model.getItems doesn't seem to work DynamoDB limits the number of items you can get to 100 or 1MB of data for a single request. Vogels automatically handles splitting up into multiple requests to load all items.

The reason being is that when it detects that the request was capped (by UnprocessedKeys not being null/empty), it doesn't construct the subsequence request properly, resulting in: Missing required key 'RequestItems' in params

I believe this is a related Vogels issue: https://github.com/ryanfitz/vogels/issues/166

I've tracked down the source of the issue here: https://github.com/clarkie/dynogels/blob/master/lib/batch.js#L46

After we get UnprocessedKeys (due to 1mb limit), it tries to request further results automatically. However it doesn't build the subsequent request properly, as you can see here: https://github.com/clarkie/dynogels/blob/master/lib/batch.js#L13 it is supposed to set RequestItems: as the base key, whereas it's not.

This results in: Original Request:

{
  "name": "dynogels",
  "hostname": "blah",
  "pid": 3652,
  "model": "Model_name",
  "level": 30,
  "params": {
    "RequestItems": {
      "table_name": {
        "Keys": [ { keys_go_here: ''}  ]
      }
    }
  },
  "msg": "dynogels BATCHGET request",
  "time": "2017-03-22T03:55:35.890Z",
  "v": 0
}

subsequent ones:

{
  "name": "dynogels",
  "hostname": "blah",
  "pid": 3652,
  "model": "Model_name",
  "level": 30,
  "params": {
      "table_name": {
        "Keys": [ { keys_go_here: ''}  ]
      }
  },
  "msg": "dynogels BATCHGET request",
  "time": "2017-03-22T03:55:35.890Z",
  "v": 0
}

note the missing RequestItems in the second one.