Reduce latency in aggregation

[vkuznet]

We need to speed-up aggregation process and reduce latency in agg. record creation for web UI. The following steps should be done:

• change migration crontab to write more often to HDFS
• add hash key into agg docs
• add hash index into MongoDB
• change insert to upsert for MongoDB agg doc insertion
• change interval of agg docs generation and insertion into MongoDB. I think it is doable to reduce latency from 1day to 1hour.

[vkuznet]

Changes to MongoDB upload went into https://github.com/dmwm/WMArchive/pull/294 Now all records have a hash key produced as md5 hash from scope content of the record. This hash is used to verify existence of agg. records in MongoDB aggregated.performance collection. And, allows to run aggregate_performance script which injects records into aggregated.performance collection any number of times during a day.

Once changes will be tested on vocms013 I'll modify cronjobs to run more often during a day and we'll get updates quickly and reduce latencies on wmarchive web ui.

[vkuznet]

I modified MongoIO layer to include current date date in #296

[vkuznet]

Changes for cronjobs went into https://github.com/dmwm/deployment/pull/423

[vkuznet]

Changes for WMArchive version went into https://github.com/cms-sw/cmsdist/pull/2750

[vkuznet]

Done and in production