SEVERE error when creating cloud watch metrics

[xmark]

I have just cloned the repo as of yesterday, and am trying to run the JAR file directly via command line on my local laptop (for testing).

I have configured the JSON config file to aggregate on MINNUTE, HOUR and FOREVER (see below)

[ { "namespace":"TestJsonConfigApp", "labelItems":["EmailAddress"], "type":"COUNT", "timeHorizons":["MINUTE","HOUR", "FOREVER"], "dataExtractor":"JSON", "dateItem":"EventDateTime", "tableName":"TestTable", "emitMetrics" : true, "readIOPS":20, "writeIOPS":40, "IDataStore":"com.amazonaws.services.kinesis.aggregators.datastore.DevNullDataStore" } ]

I have then added some events through the KCL libraries (which works fine), then I can see errors in the aggregator logs saying:

Unable to Parse Date Value H-1970-01-18 02:00:00

I can see this value in the DynamoDB table, but it looks like the data that the aggregator adds, not from me... what am I doing wrong?

The same happens for the minute aggregations, such as m-1970-01-18 ...

[IanMeyers]

Hello,

Sorry to hear you are having this problem. This date convention was scrapped in a really old version of the aggregators package, so I'm concerned that someone you've got an out of date repo. Can you confirm that the jar you are using is .9.2.6.9?

Thx,

Ian

[xmark]

Actually I am using the command line arguments that are on the main page on github, which actually invoke the AmazonKinesisAggregators.jar-complete.jar JAR and not the amazon-kinesis-aggregators-.9.2.6.9.jar. However, I can confirm that when I cloned the repo, I DO have the 9.2.6.9 JAR...

This is my command I am running: java -cp AmazonKinesisAggregators.jar-complete.jar -Dconfig-file-path=json-aggregator.json -Dstream-name=TestStream -Dapplication-name=MyApp -Dregion=ap-southeast-2 com.amazonaws.services.kinesis.aggregators.consumer.AggregatorConsumer

[IanMeyers]

OK - Can you paste an example of the input event?

[xmark]

are you talking about the input even from the "producer" application (i.e. the payload send to Kinesis) or the record in the DynamoDB table?

If you want to replicate this bug, just run the command with what I pasted earlier, the run the KCL producers sample console app, and put some messages into Kinesis.

[IanMeyers]

Hi,

Yeah can repro now - couldnt before. Will have a fix to you shortly.

Ian

[xmark]

Woot, thanks Ian!!

[IanMeyers]

Fixed in .9.2.7 - now available in the dist folder.

[IanMeyers]

Also fixed an issue whereby the DevNullDataStore was not used. Now you should NOT get a DynamoDB table created with the configuration you've cited above.

[xmark]

Thanks Ian, I will be re-trying this very soon

[xmark]

Hi Ian, doesnt look like this worked... sorry, I pulled the latest and got the new distro's but now I get an error:

Sep 17, 2015 11:05:14 PM com.amazonaws.services.kinesis.aggregators.metrics.CloudWatchMetricsEmitter emit SEVERE: Unable to Parse Date Value -

The command I am running is this: java -cp AmazonKinesisAggregators.jar-complete.jar -Dconfig-file-path=json-aggregator.json -Dstream-name=MyStreamName -Dapplication-name=MyApp -Dregion=ap-southeast-2 com.amazonaws.services.kinesis.aggregators.consumer.AggregatorConsumer

It also doesnt look like its creating the tableName that I specified in the config

There is a table called MyApp (from the parameters above), and it has 2 rows, one for each shard in Kinesis (i assume).

Here is a bigger dump of the log when I add events to the Kinesis stream:

INFO: Aggregating 13 records for Kinesis Shard shardId-000000000001 Sep 17, 2015 11:11:54 PM com.amazonaws.services.kinesis.aggregators.StreamAggregator logInfo INFO: [shardId-000000000001] Aggregation Complete - 13 Records and 13 Elements in 15 ms Sep 17, 2015 11:11:54 PM com.amazonaws.services.kinesis.aggregators.cache.AggregateCache logInfo INFO: [shardId-000000000001] Cache Flushed 3 modifications in 0ms Sep 17, 2015 11:11:54 PM com.amazonaws.services.kinesis.aggregators.metrics.CloudWatchMetricsEmitter emit SEVERE: Unable to Parse Date Value -

[IanMeyers]

Ah yes I see - the FOREVER metric was not formatted properly. This should be fixed in .9.2.7.1.

As for the table, you will have a table called MyApp, which is created by the KCL for lease tracking, but when you use the DevNullDataStore you will not have any other tables in DynamoDB.

[xmark]

OK we have some progress :)

Looks like there is a new cloud watch metric called TestTable, which is good, but it says that the Calculation is SUM when I specified COUNT in the config file?

[IanMeyers]

Hi,

Yes, that's correct. We have to SUM the eventCount, as this is the only way to add up all the eventCount entries for a period of time in CloudWatch.

Does that make sense?

Ian

[xmark]

Yeah, it makes sense, I just need to make sure I setup my reporting correctly with the config file to capture the volumes of traffic/events that I want to test under heavy load.

[IanMeyers]

Closing now