NCEAS / metadig-engine

MetaDig Engine: multi-dialect metadata assessment engine
7 stars 5 forks source link

Add quality engine health check #236

Open gothub opened 4 years ago

gothub commented 4 years ago

Add a check for the quality engine to the monitor facility used by the NCEAS compute team.

The check should determine that requested quality reports are being fetched and that quality reports are being created for the most recent content.

This health check is a bit different than an audit (described in #176), as this doesn't verify content or initiate any action, it just checks that the system is up and responding.

gothub commented 4 years ago

The 'k8s best practices' way to implement checks is with readiness and liveness probes as outlined herehttps://cloud.google.com/blog/products/gcp/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes.

This may be overkill, but this method can also be plugged into the container startup so that k8s knows exactly when to make a pod available.

One way to use these probes is to have a metadig-engine REST endpoint (monitor?) that tells the controller to send out all necessary probes, wait for a certain time and then report back with the results.

Probes/checks could be enabled for: