Size and monitor logsearch queue

[jmcarp]

The redis queue in the logsearch release has a default congestion threshold of 1m: https://github.com/cloudfoundry-community/logsearch-boshrelease/blob/8cb7e9391027012b439246f67186fdc0d9e3310b/jobs/ingestor_syslog/spec#L65-L67. In our case, that threshold is far less than the capacity of the box, so we should raise it to a reasonable level.

• [x] Ensure queue vm has memory / disk capacity for X days of logs
• [x] Ensure ingestor congestion threshold can store X days of logs (or disable threshold)
• [x] Alert when queue is at Y% of congestion threshold (or memory usage)

As an aside, the default congestion threshold in the logsearch release is different from the default threshold in vanilla logstash, which IMO is confusing. I'm thinking of submitting a patch to align the defaults with logstash. WDYT?

[rogeruiz]

Working on operator docs for back-of-the-envelope calculations for VM memory and disk space. 18F/cg-site#708

[cnelson]

Queue sized has been increased in production, but in our testing in staging, we did not see alerts when redis was facing memory contention.

Leaving this story in-progress until we can verify that we are alerted when the ingestors or queue are running out of RAM or DISK.

[cnelson]

@jmcarp @roger I'm calling this done. If you agree can one of you review and close this?

I don't know if we should we get https://github.com/18F/cg-site/pull/708 merged before we close this.

[jmcarp]

I accept!