search

silvermine / dynamodb-table-sync

MIT License
45 stars 15 forks source link

ERROR: unprocessed keys in batchGet #6

Closed er1c closed 5 years ago

er1c commented 5 years ago

I haven't dug into this, but I got this error on my first run at the end (and at the end of a 2nd run too)

https://github.com/silvermine/dynamodb-table-sync/blob/master/src/Synchronizer.js#L527


    at /c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/src/Synchronizer.js:527:22
    at _fulfilled (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:854:54)
    at self.promiseDispatch.done (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:883:30)
    at Promise.promise.promiseDispatch (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:816:13)
    at /c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:624:44
    at runSingle (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:137:13)
    at flush (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:125:13)
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9) Error: ERROR: unprocessed keys in batchGet
    at /c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/src/Synchronizer.js:527:22
    at _fulfilled (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:854:54)
    at self.promiseDispatch.done (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:883:30)
    at Promise.promise.promiseDispatch (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:816:13)
    at /c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:624:44
    at runSingle (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:137:13)
    at flush (/c/Users/Eric Peters/.npm-packages/lib/node_modules/silvermine-dynamodb-table-sync/node_modules/q/q.js:125:13)
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9)```
jthomerson commented 5 years ago

Yeah, basically that will happen if the tool is reading too fast for your table. We aren't doing re-querying for when we do a query and DynamoDB says "hey, that's too much data, you need to slow down ... here's some data, and here's some keys I didn't process that you need to try again for".

The way around this is to play with the CLI flags a bit and slow down the reads. Try --batch-read-limit and --scan-limit. (Or turn on DynamoDB on-demand for the table)

er1c commented 5 years ago

@jthomerson good catch, forgot to enable ondemand on the new table, I'll update and verify