Add health probes to API server

[sayan-biswas]

Changes

Fixes #280 Fixes #414

This PR adds gRPC and REST health probe endpoints to the API server and configures the liveliness, readiness and startup probe for the API server deployment. Also adds retries for connecting to database to prevent unnecessary pod restarts.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

• [x] Has Docs included if any changes are user facing
• [ ] Has Tests included if any functionality added or changed
• [x] Tested your changes locally (if this is a code change)
• [x] Follows the commit message standard
• [x] Meets the Tekton contributor standards (including functionality, content, code)
• [x] Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
• [ ] Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
• [ ] Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

- api: Add healthz endpoint to enable liveness, readiness, and startup probes for the api-server deployment. Default configuration uses HTTP, but gRPC can also be used.
- api: Poll up to two minutes when establishing the database connection.

[tekton-robot]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

[sayan-biswas]

/kind feature

[sayan-biswas]

/ok-to-test

[adambkaplan]

/approve

Accepting as a feature.

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adambkaplan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/tektoncd/results/blob/main/OWNERS)~~ [adambkaplan] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[adambkaplan]

Proposing updated release note:

- api: Add healthz endpoint to enable liveness, readiness, and startup probes for the apiserver deployment.
- api: Poll up to two minutes when establishing the database connection.

[enarha]

The change is LGTM, but lets give a chance to other people to review.

[alan-ghelardi]

/lgtm

[enarha]

/test pull-tekton-results-integration-tests