Low performance

[kazjote]

I just deployed Kinesis Tee (kinesis_tee_0.1.0.zip) function as a consumer of a Kinesis stream with quite some traffic. My batch size is 100. There is no transformation and JavaScript filter function is completely trivial:

function filter(record) {
    var derivedEventNameCol = 126;

    return (record.split('\t')[derivedEventNameCol] !== "application_error");
}

Since the deployment, function is getting more and more behind (according to lambda IteratorAge metric).

CloudWatch Logs report also unexpectedly high execution time:

REPORT RequestId: bc87b6d1-08be-4981-be6e-7cb0652d6bd9 Duration: 4181.63 ms Billed Duration: 4200 ms Memory Size: 512 MB Max Memory Used: 153 MB 

All invocations are successful and data is written correctly to target stream. Target stream is not throttled.

As far as I understand, the problem is that Kinesis Tee writes records sequentially to target stream record by record

Writes should be done either:

1. in a batch - probably the best solution as API allows to write up to 500 record in a single batch. This make it cost efficient (less API calls)
2. in parallel - multiple writes at the same time

What do you think about it?

[alexanderdean]

Thanks for flagging - the batch appropriate is probably the most straightforward, scheduling...