integration tests leak goroutines

[MHBauer]

Bug Report

What happened: About 7000 goroutines are running at end of a test run.

This is with #2667, but I expect it exists on master

What you expected to happen: Each individual run should clean up after itself. There should be as close to 0 running goroutines by the end. Roughly 1 to 3.

How to reproduce it (as minimally and precisely as possible): drop this into controller_flow_instance_test.go

func TestMain(m *testing.M) {
    // call flag.Parse() here if TestMain uses flags
    exit := m.Run()

    fmt.Println(runtime.NumGoroutine())
    pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)

    os.Exit(exit)
}

and run go test -tags=integration -v -count 1 -failfast ./pkg/controller

see output similar to

7408
goroutine profile: total 7372
3656 @ 0x1032da0 0x1007af8 0x1007ace 0x10077fb 0x1b1f290 0x10602c1
#   0x1b1f28f   github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop+0xdf /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue/queue.go:198

3628 @ 0x1032da0 0x104270b 0x1b1ccc4 0x10602c1
#   0x1b1ccc3   github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x343 /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:209

86 @ 0x1032da0 0x1007af8 0x1007ace 0x10077fb 0x1e02024 0x10602c1
#   0x1e02023   github.com/kubernetes-sigs/service-catalog/pkg/controller_test.newControllerTest.func1+0x53 /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/case_test.go:128

1 @ 0x1032da0 0x1007af8 0x1007ace 0x10077fb 0x132864b 0x10602c1
#   0x132864a   github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/klog.(*loggingT).flushDaemon+0x8a  /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/klog/klog.go:951

1 @ 0x1147bc5 0x11479e0 0x114460a 0x1dfa7e9 0x1e09565 0x10329ce 0x10602c1
#   0x1147bc4   runtime/pprof.writeRuntimeProfile+0x94                      /Users/mhb/.go/src/runtime/pprof/pprof.go:708
#   0x11479df   runtime/pprof.writeGoroutine+0x9f                       /Users/mhb/.go/src/runtime/pprof/pprof.go:670
#   0x1144609   runtime/pprof.(*Profile).WriteTo+0x3d9                      /Users/mhb/.go/src/runtime/pprof/pprof.go:329
#   0x1dfa7e8   github.com/kubernetes-sigs/service-catalog/pkg/controller_test.TestMain+0xe8    /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller_flow_instance_test.go:41
#   0x1e09564   main.main+0x134                                 _testmain.go:504
#   0x10329cd   runtime.main+0x21d                              /Users/mhb/.go/src/runtime/proc.go:203

for count 2

14767
goroutine profile: total 14724
7284 @ 0x1032da0 0x1007af8 0x1007ace 0x10077fb 0x1b1f290 0x10602c1
#   0x1b1f28f   github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop+0xdf /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue/queue.go:198

7256 @ 0x1032da0 0x104270b 0x1b1ccc4 0x10602c1
#   0x1b1ccc3   github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop+0x343 /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:209

172 @ 0x1032da0 0x1007af8 0x1007ace 0x10077fb 0x1e02024 0x10602c1
#   0x1e02023   github.com/kubernetes-sigs/service-catalog/pkg/controller_test.newControllerTest.func1+0x53 /Users/mhb/go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/case_test.go:128

Anything else we need to know?: similar to #434 #1649 #1671

I don't know how tags work, I'm assuming tags=integration is running only the stuff with the +build integration tag, but if not the issue might be in the unit tests.

Environment: one of these is the fake recorder https://github.com/kubernetes-sigs/service-catalog/blob/master/pkg/controller/case_test.go#L118-L123

I'll let you find the two others.

[mszostok]

@MHBauer really nice catch! definitely, we will need to dig into that. Fortunately, this is not affecting our end-users

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/service-catalog/issues/2698#issuecomment-581182705): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[mszostok]

/reopen /remove-lifecycle rotten

[k8s-ci-robot]

@mszostok: Reopened this issue.

In response to [this](https://github.com/kubernetes-sigs/service-catalog/issues/2698#issuecomment-633744464): >/reopen >/remove-lifecycle rotten Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[MHBauer]

/lifecycle freeze

[MHBauer]

/lifecycle frozen

[mrbobbytables]

This project is being archived, closing open issues and PRs. Please see this PR for more information: https://github.com/kubernetes/community/pull/6632