Implement new Bigtable timeout & retry settings

[sming]

Use Case Resolved: Heroic pipeline instability and poor user experience for Grafana

• I am a Heroic developer
• who wants to armour up Heroic so that we get fewer alerts
• so that we can focus on features

Plan

• look at the spotify-bigtable-client repo for "inspiration"
• don't implement a whole new layer of configuration. Focus on the 2 or 3 settings we're immediately interested in.

Notes for Reviewer

• only read requests are affected (i.e. Heroic API but not metric Ingestion)
• a maxMutateRowTimeout is implemented but will be deployed with the default setting value so as not to change Ingestion at all. This is done for development efficiency reasons i.e. it would have been foolish not to add support for it whilst I was changing the exact same files in the exact same places. Here is the heroic-gke PR that puts mutateRpcTimeoutMs back to 10 minutes.

[codecov[bot]]

Codecov Report

Merging #733 (e380555) into master (2e63167) will increase coverage by 0.10%. The diff coverage is 89.13%.

@@             Coverage Diff              @@
##             master     #733      +/-   ##
============================================
+ Coverage     55.09%   55.20%   +0.10%     
- Complexity     3163     3175      +12     
============================================
  Files           749      751       +2     
  Lines         20427    20544     +117     
  Branches       1341     1343       +2     
============================================
+ Hits          11254    11341      +87     
- Misses         8684     8716      +32     
+ Partials        489      487       -2     

Impacted Files	Coverage Δ	Complexity Δ
...m/spotify/heroic/http/tracing/OpenCensusUtils.java	0.00% <ø> (ø)	0.00 <0.00> (ø)
...c/test/AbstractMetadataBackendIndexResourceIT.java	94.47% <0.00%> (ø)	29.00 <0.00> (ø)
...ic/analytics/bigtable/BigtableAnalyticsModule.java	65.07% <0.00%> (ø)	9.00 <0.00> (ø)
...heroic/suggest/elasticsearch/SuggestBackendKV.java	82.22% <ø> (+0.74%)	79.00 <0.00> (+2.00)
...com/spotify/heroic/metric/MetricManagerModule.java	71.31% <23.80%> (-2.80%)	18.00 <0.00> (ø)
...m/spotify/heroic/metric/consts/ApiQueryConsts.java	66.66% <66.66%> (ø)	1.00 <1.00> (?)
...spotify/heroic/metric/MetricsConnectionSettings.kt	69.23% <69.23%> (ø)	2.00 <2.00> (?)
...potify/heroic/metric/bigtable/BigtableBackend.java	85.76% <83.33%> (-0.06%)	61.00 <1.00> (ø)
...y/heroic/metric/bigtable/BigtableMetricModule.java	81.51% <87.50%> (-4.35%)	6.00 <0.00> (-1.00)
.../spotify/heroic/test/AbstractSuggestBackendIT.java	99.14% <98.60%> (-0.36%)	53.00 <30.00> (+6.00)	:arrow_down:
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2e63167...e380555. Read the comment docs.