Summary
debugging with @joshuasing , we were noticing in pprof output that we have many goroutines hanging at the request limiter, we then realized that we don't stop waiting if the request's context is done. therefore we (very likely) have a bunch of goroutines with "done" contexts that are hanging and hogging resources
Changes
when using the request limiter, ensure that we don't continue to wait on a context that is done
Summary debugging with @joshuasing , we were noticing in
pprofoutput that we have many goroutines hanging at the request limiter, we then realized that we don't stop waiting if the request's context is done. therefore we (very likely) have a bunch of goroutines with "done" contexts that are hanging and hogging resourcesChanges when using the request limiter, ensure that we don't continue to wait on a context that is done