twitchscience / kinsumer

Native Go consumer for AWS Kinesis streams.
Other
134 stars 35 forks source link

Data race during Stop() #19

Closed dcelasun closed 5 years ago

dcelasun commented 5 years ago

This one is very simple to reproduce with noopkinsumer, but to get it to build at all with the latest AWS SDK, a small fix is necessary:

git diff .
diff --git a/cmd/noopkinsumer/main.go b/cmd/noopkinsumer/main.go
index 984c5ba..780e869 100644
--- a/cmd/noopkinsumer/main.go
+++ b/cmd/noopkinsumer/main.go
@@ -57,7 +57,9 @@ func initKinsumer() {
        }

        config := kinsumer.NewConfig().WithStats(stats)
-       session := session.Must(session.NewSession(aws.NewConfig()))
+       ac := aws.NewConfig()
+       ac.Region = aws.String("us-east-1")
+       session := session.Must(session.NewSession(ac))

        // kinsumer needs a way to differentiate between running clients, generally you want to use information
        // about the machine it is running on like ip. For this example we'll use a uuid

Then simply build the binary with the -race flag:

$ cd cmd/noopkinsumer
$ go build -race .
$ ./noopkinsumer -stream example -createTables

and send a quit signal with CTRL-C.

Results:

./noopkinsumer -stream anlytics-testing-stream -createTables
^C2018/11/21 11:42:12 Total records consumed 123
==================
WARNING: DATA RACE
Write at 0x00c420092a50 by goroutine 48:
  github.com/twitchscience/kinsumer.(*Kinsumer).Run.func1.1()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/kinsumer.go:362 +0x22e
  github.com/twitchscience/kinsumer.(*Kinsumer).Run.func1()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/kinsumer.go:403 +0xa5f

Previous read at 0x00c420092a50 by goroutine 47:
  github.com/twitchscience/kinsumer.(*Kinsumer).becomeLeader.func1()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/leader.go:79 +0x226

Goroutine 48 (running) created at:
  github.com/twitchscience/kinsumer.(*Kinsumer).Run()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/kinsumer.go:350 +0x4c7
  main.runKinsumer()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/cmd/noopkinsumer/main.go:82 +0x4e
  main.main()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/cmd/noopkinsumer/main.go:136 +0x86

Goroutine 47 (running) created at:
  github.com/twitchscience/kinsumer.(*Kinsumer).becomeLeader()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/leader.go:43 +0xef
  github.com/twitchscience/kinsumer.(*Kinsumer).refreshShards()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/kinsumer.go:154 +0x7d4
  github.com/twitchscience/kinsumer.(*Kinsumer).Run()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/kinsumer.go:340 +0x158
  main.runKinsumer()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/cmd/noopkinsumer/main.go:82 +0x4e
  main.main()
      /home/dcelasun/go/src/github.com/twitchscience/kinsumer/cmd/noopkinsumer/main.go:136 +0x86
==================
Found 1 data race(s)