DynamoDB: General backlog

About

Coming from GH-226, GH-247, GH-252, GH-263, and others, there are a few backlog items to be resolved.

Items

General

[ ] GH-301
[ ] GH-232
[ ] Transformations. That's another item transformation idea picked up from an example program.
```
if 'id' in item and not isinstance(item['id'], str):
   item['id'] = str(item['id'])
```
Please advise if this is sensible in all situations, or if it's just a special case.

Full Load

[x] Processing records in batches. https://github.com/crate/cratedb-toolkit/pull/252
[ ] Scan by query instead of doing a full scan.
[x] Improve error handling by using BulkProcessor, like MongoDB I/O is doing it. https://github.com/crate/cratedb-toolkit/pull/262
[ ] Using DynamoDB Scan+Pagination needs a software test.
[ ] Decouple ingress pagination from egress batching?
[ ] Implement Parallel scan.

Change Stream

[ ] Improve error handling of processor Lambda by storing invalid records into a dead letter box.
[ ] Consume from AWS Kinesis Stream ARN instead of stream name only, to make it work across organization boundaries, for example through connectivity using AWS PrivateLink. Via GH-247.

Manual Testing

Other than having solid software tests, which are included into corresponding PRs, the DynamoDB I/O subsystem must be manually tested for semantic soundness and good usability, specifically but not exclusively related to full-load -> cdc convergence.

In the best possible scenario, this would happen on real-world data instead of synthesized or other data coming from testing datasets, because those are mostly uniform and homogeneous. In this spirit, the former is crucial for discovering edge case situations, and hopefully improving them.

crate / cratedb-toolkit