move log storage off reconciler thread (again)

[gabemontero]

Changes

There was a great disparity between performance once we applied sufficient concurrency on production scale systems with log storage on the reconciler thread.

After a couple of weeks of analysis between myself and @pmacik, the primary scaling bottleneck was at the golang http level, specifically the http2 Body.Read(), at https://github.com/tektoncd/results/blob/ca1ca870efebf9e122684134a9d38862c0be2375/vendor/google.golang.org/grpc/internal/transport/handler_server.go#L420

For the time being then, that has forced us to move log storage back off of the log storage thread.

We will circle back to retry on recoverable errors once the Log status changes are in.

Details on the changes in this PR include:

• moved all of log storage back onto a separate goroutine, off the dynamic reconciler thread
• removed any unbuffered channels, used only buffered channels, to unblock golang GC
• we employ the backgrond context with cancel function so no contexts are left hang, to enable cleanup of grpc client threads
• those two changes in tandem prevent the original memory leak
• we employ a configurable timeout for the dynamic reconciler context
• we employ a configurable timer to handle timeout with the background context on the actual UpdateLog calls
• we removed the goroutine dumps on canceled context as we now have a handle on the "hang"

/kind bug

@khrm @enarha PTAL

@sayan-biswas @avinal fyi / ptal if you all have cycles

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

• [ ] Has Docs included if any changes are user facing
• [x] Has Tests included if any functionality added or changed
• [x] Tested your changes locally (if this is a code change)
• [x] Follows the commit message standard
• [x] Meets the Tekton contributor standards (including functionality, content, code)
• [x] Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
• [x] Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
• [ ] Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Prior attempts to fix the watcher memory leak when log storage is enabled (originally issue #695) placed log storage on the dynamic reconciler to facilitate retry on retryable errors.  However, production level testing uncovered underlying golang/http2 performance issues which made this untenable.

This change moves log storage back off the dynamic reconciler thread, but still addresses the memory leak, as well as intermittent 'canceled context' errors during log storage through the results api server.

Two new command line arguments are enabled on the watch to facilitate timeouts on log storage:
1) dynamic_reconcile_timeout: How long the Watcher waits for the dynamic reconciler to complete before it aborts
2) update_log_timeout: How log the Watcher waits for the UpdateLog operation for storing logs to complete before it aborts.

[tekton-robot]

The following is the coverage report on the affected files. Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/logs.go	63.4%	63.2%	-0.2
pkg/watcher/reconciler/dynamic/dynamic.go	55.3%	53.0%	-2.3

[tekton-robot]

The following is the coverage report on the affected files. Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/logs.go	63.4%	63.2%	-0.2
pkg/watcher/reconciler/dynamic/dynamic.go	55.3%	54.5%	-0.8

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: avinal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/tektoncd/results/blob/main/OWNERS)~~ [avinal] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment