update_stat_aggregation issues with Cassandra 'eventually consistent' mode.

[mitchell-es]

I have set up the east and west clusters to replicate the data via this: CREATE KEYSPACE esmond WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '2' };

While running my data migration script I began seeing this error which did not happen before setting the replication_factor to 2. Note that it can fail on both the 'min' and 'max' case at line 448. esmond.log.0.bz2:Sep 4 23:59:16 localhost emigrate [93317] Traceback (most recent call last): File "./mover.py", line 89, in MigrateViaPool cpersister.store(pr) File "/usr/home/mitchell/emigrate/src/esmond/esmond/persist.py", line 597, in store self.generate_aggregations(raw_data, oidset.aggregates) File "/usr/home/mitchell/emigrate/src/esmond/esmond/persist.py", line 907, in generate_aggregations self._agg_timestamp(data, freq), freq*1000) File "/usr/home/mitchell/emigrate/src/esmond/esmond/cassandra.py", line 443, in update_stat_aggregation elif agg.val > ret['max']: KeyError: 'max'

What I tried but did not help is changes along the following lines. The errors are continuing so I'm not sure exactly what causes this behavior.

@@ -393,7 +393,7 @@ class CASSANDRA_DB(object):

name key that is not 'val' - this will be used by the query interface

     # to generate the averages.  Both values are counter types.
     self.aggs.insert(agg.get_key(), 

• {agg.ts_to_jstime(): {'val': agg.val, str(agg.base_freq): 1}})
• {agg.ts_to_jstime(): {'val': agg.val, str(agg.base_freq): 1}},ConsistencyLevel.ALL)

[mitchell-es]

Here's an example CQL query showing how the values in a single super column appear as multiple rows:

cqlsh:esmond> select * from stat_aggregations where key='snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014' limit 8; 

 key                                                           | column1       | column2        | value
---------------------------------------------------------------+---------------+----------------+---------------
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409353200000 |       0x6d6178 |          4608
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409353200000 | 0x6d61785f7473 | 1409355549000
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409353200000 |       0x6d696e |          3328
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409353200000 | 0x6d696e5f7473 | 1409354405000
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409356800000 |       0x6d6178 |          4608
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409356800000 | 0x6d61785f7473 | 1409356879000
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409356800000 |       0x6d696e |          3328
 snmp:star-asw1:FastPollHC:ifHCInOctets:ge-0/0/20:3600000:2014 | 1409356800000 | 0x6d696e5f7473 | 1409360127000

(8 rows)

cqlsh:esmond>