Jbruno/remove subscription from cluster update

[JeanetteBruno]

eliminate subscribe fetcher/cluster/cluster_state/groupcache now operate as follows: fetcher

• when cron fetcher is started it returns the channel it will use to send the latest node list to cluster. This will be a buffered channel preventing cluster from blocking fetcher

cluster

• when cluster is created it creates a cron fetcher, getting the (buffered) channel that will contain fetched node lists
• cluster processing loop . empties the fetched node list channel (or pulls at most buffer size entries from the channel) and either . converts the node list to a list of NodeUpdates and puts the node udpate events on the (buffered) node update channel (sending them to cluster_state) or . sends the list of current nodes on a nodes request channel, when the nodes are requested (by apiserver)
• we are using buffered channels to prevent cluster blocking fetcher and to prevent cluster_state blocking cluster

cluster_state

• update() , pulls all []NodeUpdate lists off the nodes updates channel (shared with cluster), and updates the cluster

groupcache

• sends a nodes request channel to cluster
• pulls the nodes from the nodes request channel
• updates peers with this list of nodes

Also added were metrics for tracking the various cluster update frequencies, fetch duration and fetch errors.

Also updated a number of fmt.Errorf() statements per go's conventions (don't start error messages with capital letter, nor ending punctuation)

[codecov-commenter]

Codecov Report

Merging #556 (ed79bca) into master (cc7ebe6) will decrease coverage by 0.15%. The diff coverage is 64.00%.

:exclamation: Current head ed79bca differs from pull request most recent head a888036. Consider uploading reports for the commit a888036 to get more accurate results

@@            Coverage Diff             @@
##           master     #556      +/-   ##
==========================================
- Coverage   55.25%   55.10%   -0.16%     
==========================================
  Files         120      119       -1     
  Lines        8616     8591      -25     
==========================================
- Hits         4761     4734      -27     
+ Misses       3343     3342       -1     
- Partials      512      515       +3     

Impacted Files	Coverage Δ
cloud/cluster/state.go	100.00% <ø> (+6.66%)	:arrow_up:
scheduler/scheduler/config/config.go	50.00% <0.00%> (-2.55%)	:arrow_down:
scheduler/server/stateful_scheduler.go	63.95% <16.66%> (+0.99%)	:arrow_up:
scheduler/server/cluster_state.go	82.35% <70.58%> (-3.44%)	:arrow_down:
cloud/cluster/cluster.go	76.66% <76.66%> (-15.93%)	:arrow_down:
cloud/cluster/fetch_cron.go	85.18% <84.61%> (+12.45%)	:arrow_up:
cloud/cluster/node.go	56.52% <0.00%> (ø)
scheduler/domain/generators.go	71.64% <0.00%> (+4.47%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update cc7ebe6...a888036. Read the comment docs.