search

jcelliott / turnpike

Go implementation of a WAMP (Web Application Messaging Protocol) client and router
MIT License
258 stars 88 forks source link

fatal error: bad map state #105

Closed glycerine closed 8 years ago

glycerine commented 9 years ago

go1.5.1 on osx. turnpike at c7792e9fac4db78b251de07ecd4249d6e7880ce0

trying to do a little rpc benchmark, go panics on bad state of map. I suspect the Client.listeners map at turnpike/client.go:304 (and everywhere else it is modified) probably needs to be protected by a mutex, as it looks like it got corrupted with a bunch of calls in progress. This happens after a couple hundred calls in on the same client-to-server socket.

partial stack trace, showing the source code and line numbers where the panic originates:

...
2015/10/18 18:03:13 client.go:302: register listener: 3862323585691745
2015/10/18 18:03:13 client.go:302: register listener: 2726553413377910
2015/10/18 18:03:13 client.go:302: register listener: 5791692451129500
2015/10/18 18:03:13 client.go:302: register listener: 4772997322193826
fatal error: bad map state
2015/10/18 18:03:13 client.go:302: register listener: 1553043796640561
2015/10/18 18:03:13 client.go:302: register listener: 2705276085876327
cli: error setting alarm: 'websocket: close sent'. call result = '%!s(*turnpike.Result=<nil>)'.
cli: error setting alarm: 'websocket: close sent'. call result = '%!s(*turnpike.Result=<nil>)'.

goroutine 210 [running]:
runtime.throw(0x10ef9d950, 0xd)
    /usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc8204d9b60 sp=0xc8204d9b48
runtime.evacuate(0x10edeaf80, 0xc8200ec360, 0xf)
    /usr/local/go/src/runtime/hashmap.go:825 +0x3b1 fp=0xc8204d9c20 sp=0xc8204d9b60
runtime.growWork(0x10edeaf80, 0xc8200ec360, 0x61)
    /usr/local/go/src/runtime/hashmap.go:795 +0x83 fp=0xc8204d9c40 sp=0xc8204d9c20
runtime.mapassign1(0x10edeaf80, 0xc8200ec360, 0xc8204d9d10, 0xc8204d9d18)
    /usr/local/go/src/runtime/hashmap.go:433 +0x176 fp=0xc8204d9ce8 sp=0xc8204d9c40
vendor/github.com/glycerine/turnpike.(*Client).registerListener(0xc820010be0, 0x9afc92faa5f76)
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/turnpike/client.go:304 +0x1ab fp=0xc8204d9d60 sp=0xc8204d9ce8
vendor/github.com/glycerine/turnpike.(*Client).Call(0xc820010be0, 0x10ef9d250, 0x9, 0xc820500e10, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0)
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/turnpike/client.go:488 +0x58 fp=0xc8204d9e78 sp=0xc8204d9d60
main.client_main()
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/wamp-client.go:52 +0x488 fp=0xc8204d9fa0 sp=0xc8204d9e78
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8204d9fa8 sp=0xc8204d9fa0
created by main.Cli
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:51 +0x28a
glycerine commented 9 years ago

Indeed, I find that running "go test -v -race" in the main turnpike/ directory reveals a multitude of data races waiting to be fixed.

~/go/src/github.com/glycerine/turnpike$ go test -v -race
=== RUN   TestSubscribe

  Subscribing to a topic 
    The subscriber should have received a SUBSCRIBED message ✔
    The broker should have created the subscription ✔✔

3 total assertions

--- PASS: TestSubscribe (0.00s)
=== RUN   TestUnsubscribe

  Unsubscribing from a topic 
    The peer should have received an UNSUBSCRIBED message ✔
    The broker should have removed the subscription ✔✔

6 total assertions

--- PASS: TestUnsubscribe (0.00s)
=== RUN   TestJoinRealm

  Given a server accepting client connections ==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/glycerine/turnpike.(*localClient).onJoin()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:78 +0xaf
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:87 +0x116

Previous write by goroutine 9:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:56 +0x322
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 10 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 9 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/glycerine/turnpike.(*Client).Publish()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:479 +0x1cf
  github.com/glycerine/turnpike.(*localClient).onJoin()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:78 +0xff
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:87 +0x116

Previous write by goroutine 9:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:56 +0x1c3
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 10 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 9 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:113 +0x7ee

Previous write by goroutine 9:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:64 +0x8a1
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 10 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 9 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:126 +0xf48

Previous write by goroutine 9:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:67 +0x973
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 10 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 9 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:136 +0x16ff

Previous write by goroutine 9:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:58 +0x4d7
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 10 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 9 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Read by goroutine 10:
  github.com/glycerine/turnpike.(*defaultBroker).Publish()
      /Users/jaten/go/src/github.com/glycerine/turnpike/broker.go:41 +0x147
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:136 +0x1790

Previous write by goroutine 9:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:58 +0x41e
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 10 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealm.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:33 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:40 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 9 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================

    A client should be able to succesfully join a realm ✔

7 total assertions

--- PASS: TestJoinRealm (0.86s)
=== RUN   TestJoinRealmWithAuth

  Given a server accepting client connections 
    A client should be able to successfully authenticate and join a realm ==================
WARNING: DATA RACE
Read by goroutine 17:
  github.com/glycerine/turnpike.(*localClient).onJoin()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:78 +0xaf
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:87 +0x116

Previous write by goroutine 15:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:56 +0x322
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.TestJoinRealmWithAuth.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:52 +0x16f
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealmWithAuth()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:63 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 17 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.TestJoinRealmWithAuth.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:52 +0x16f
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealmWithAuth()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:63 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 15 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Read by goroutine 16:
  github.com/glycerine/turnpike.(*localClient).onJoin()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:78 +0xaf
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:87 +0x116

Previous write by goroutine 15:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:56 +0x322
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealmWithAuth.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:49 +0x37
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealmWithAuth()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:63 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 16 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.TestJoinRealmWithAuth.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:49 +0x37
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealmWithAuth()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:63 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 15 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
==================
WARNING: DATA RACE
Write by goroutine 19:
  runtime.mapassign1()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/hashmap.go:411 +0x0
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:86 +0xe5
  github.com/glycerine/turnpike.(*defaultRouter).Accept.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x76

Previous write by goroutine 17:
  runtime.mapassign1()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/hashmap.go:411 +0x0
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:86 +0xe5

Goroutine 19 (running) created at:
  github.com/glycerine/turnpike.(*defaultRouter).Accept()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d1

Goroutine 17 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.TestJoinRealmWithAuth.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:52 +0x16f
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestJoinRealmWithAuth()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:63 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc
==================
✔

8 total assertions

--- PASS: TestJoinRealmWithAuth (0.34s)
=== RUN   TestRemoteCall

  Given two clients connected to the same server ==================
WARNING: DATA RACE
Read by goroutine 22:
  github.com/glycerine/turnpike.(*localClient).onJoin()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:78 +0xaf
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:87 +0x116

Previous write by goroutine 21:
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:56 +0x322
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.connectedTestClients()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:17 +0x29
  github.com/glycerine/turnpike.TestRemoteCall.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:68 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestRemoteCall()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:115 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 22 (running) created at:
  github.com/glycerine/turnpike.(*Realm).getPeer()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31e
  github.com/glycerine/turnpike.(*Realm).init()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:55 +0xc5
  github.com/glycerine/turnpike.(*defaultRouter).RegisterRealm()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:89 +0x169
  github.com/glycerine/turnpike.newTestRouter()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:12 +0x2be
  github.com/glycerine/turnpike.connectedTestClients()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:17 +0x29
  github.com/glycerine/turnpike.TestRemoteCall.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:68 +0x34
  github.com/smartystreets/goconvey/convey.parseAction.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x29
  github.com/smartystreets/goconvey/convey.(*context).conveyInner()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28c
  github.com/smartystreets/goconvey/convey.rootConvey.func1()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f0
  github.com/jtolds/gls._m()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x39
  github.com/jtolds/gls.markS()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x38
  github.com/jtolds/gls.addStackTag()
      /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x49
  github.com/jtolds/gls.(*ContextManager).SetValues()
      /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x609
  github.com/smartystreets/goconvey/convey.rootConvey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x546
  github.com/smartystreets/goconvey/convey.Convey()
      /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x52
  github.com/glycerine/turnpike.TestRemoteCall()
      /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:115 +0x1c5
  testing.tRunner()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc

Goroutine 21 (running) created at:
  testing.RunTests()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/glycerine/turnpike/_test/_testmain.go:102 +0x20f
==================
✔✔
    The callee unregisters an invalid method 
      And expects an error ✔==================
WARNING: DATA RACE
Write by goroutine 28:
  runtime.mapassign1()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/hashmap.go:411 +0x0
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:86 +0xe5
  github.com/glycerine/turnpike.(*defaultRouter).Accept.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x76

Previous write by goroutine 26:
  runtime.mapassign1()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/hashmap.go:411 +0x0
  github.com/glycerine/turnpike.(*Realm).handleSession()
      /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:86 +0xe5
  github.com/glycerine/turnpike.(*defaultRouter).Accept.func1()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x76

Goroutine 28 (running) created at:
  github.com/glycerine/turnpike.(*defaultRouter).Accept()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d1

Goroutine 26 (running) created at:
  github.com/glycerine/turnpike.(*defaultRouter).Accept()
      /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d1
==================
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0xa0c05]

goroutine 38 [running]:
github.com/glycerine/turnpike.(*Client).Publish(0x0, 0x96fd60, 0x14, 0xc82011c7b0, 0x1, 0x1, 0x0, 0x0, 0x0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:485 +0x245
github.com/glycerine/turnpike.(*localClient).onJoin(0xc820192080, 0x0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:78 +0x100
github.com/glycerine/turnpike.(*Realm).handleSession(0xc820192000, 0x1d290c8, 0xc82011c780, 0x12978a6feb95e1, 0x0, 0xc820066540)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:87 +0x117
created by github.com/glycerine/turnpike.(*Realm).getPeer
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31f

goroutine 1 [chan receive]:
testing.RunTests(0xa47fe0, 0xc93760, 0x19, 0x19, 0xc820000101)
    /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:562 +0xafa
testing.(*M).Run(0xc8200b9ef0, 0x64a2da63)
    /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe5
main.main()
    github.com/glycerine/turnpike/_test/_testmain.go:102 +0x210

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 35 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc82015c360, 0x1d290c8, 0xc82000e620, 0x6249f8ee24d66, 0xc82011a4b0, 0xc820066120)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
github.com/glycerine/turnpike.(*defaultRouter).Accept.func1(0xc82015c360, 0x1d290c8, 0xc82000e620, 0x6249f8ee24d66, 0xc82011a4b0, 0xc820066120, 0xc8200121e0, 0xc82000a580)
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x77
created by github.com/glycerine/turnpike.(*defaultRouter).Accept
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d2

goroutine 23 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc8200a67e0, 0x1d290c8, 0xc820144b10, 0xb2573c0ef3752, 0x0, 0xc820094840)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
created by github.com/glycerine/turnpike.(*Realm).getPeer
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31f

goroutine 51 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc82018e000, 0x1d290c8, 0xc820144e10, 0x13410533f8d004, 0xc820186030, 0xc82018a060)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
github.com/glycerine/turnpike.(*defaultRouter).Accept.func1(0xc82018e000, 0x1d290c8, 0xc820144e10, 0x13410533f8d004, 0xc820186030, 0xc82018a060, 0xc820092aa0, 0xc820081c20)
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x77
created by github.com/glycerine/turnpike.(*defaultRouter).Accept
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d2

goroutine 4 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc82015c000, 0x1d290c8, 0xc820144b80, 0x184b912181fb1e, 0xc820014540, 0xc82001e1e0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
github.com/glycerine/turnpike.(*defaultRouter).Accept.func1(0xc82015c000, 0x1d290c8, 0xc820144b80, 0x184b912181fb1e, 0xc820014540, 0xc82001e1e0, 0xc820092a00, 0xc82000a2a0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x77
created by github.com/glycerine/turnpike.(*defaultRouter).Accept
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d2

goroutine 34 [chan receive]:
github.com/glycerine/turnpike.(*Client).Receive(0xc820012190)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:212 +0xcc
created by github.com/glycerine/turnpike.(*Client).JoinRealm
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:99 +0x90e

goroutine 6 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc82015c120, 0x1d290c8, 0xc82000e540, 0x198508a2d4d5b1, 0x0, 0xc82001e420)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
created by github.com/glycerine/turnpike.(*Realm).getPeer
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31f

goroutine 7 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc82015c240, 0x1d290c8, 0xc82000e5b0, 0x9058b39f5a221, 0x0, 0xc82001e540)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
created by github.com/glycerine/turnpike.(*Realm).getPeer
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31f

goroutine 26 [runnable]:
github.com/glycerine/turnpike.(*defaultRouter).getTestPeer(0xc8201181e0, 0x0, 0x0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:182 +0x50
github.com/glycerine/turnpike.connectedTestClients(0x3f9aaa, 0x2e)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:18 +0x3c
github.com/glycerine/turnpike.TestRemoteCall.func1()
    /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:68 +0x35
github.com/smartystreets/goconvey/convey.parseAction.func1(0x1d28e20, 0xc820094a20)
    /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/discovery.go:80 +0x2a
github.com/smartystreets/goconvey/convey.(*context).conveyInner(0xc820094a20, 0x9a2e60, 0x2e, 0xc820144ca0)
    /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:261 +0x28d
github.com/smartystreets/goconvey/convey.rootConvey.func1()
    /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:110 +0x1f1
github.com/jtolds/gls._m(0x0, 0xc820081b40)
    /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:70 +0x3a
github.com/jtolds/gls.markS(0x0, 0xc820081b40)
    /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:21 +0x39
github.com/jtolds/gls.addStackTag(0x0, 0xc820081b40)
    /Users/jaten/go/src/github.com/jtolds/gls/stack_tags.go:18 +0x4a
github.com/jtolds/gls.(*ContextManager).SetValues(0xc820081120, 0xc820142cf0, 0xc820081b40)
    /Users/jaten/go/src/github.com/jtolds/gls/context.go:98 +0x60a
github.com/smartystreets/goconvey/convey.rootConvey(0xc82005df20, 0x3, 0x3)
    /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/context.go:113 +0x547
github.com/smartystreets/goconvey/convey.Convey(0xc82005df20, 0x3, 0x3)
    /Users/jaten/go/src/github.com/smartystreets/goconvey/convey/doc.go:75 +0x53
github.com/glycerine/turnpike.TestRemoteCall(0xc8200a6900)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client_test.go:115 +0x1c6
testing.tRunner(0xc8200a6900, 0xc937c0)
    /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdd
created by testing.RunTests
    /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa4

goroutine 50 [chan receive]:
github.com/glycerine/turnpike.(*Client).Receive(0xc820012410)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:212 +0xcc
created by github.com/glycerine/turnpike.(*Client).joinRealmCRA
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:146 +0x1200

goroutine 27 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc8200a6990, 0x1d290c8, 0xc820144d80, 0x16fa2cedcbd243, 0x0, 0xc820094b40)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
created by github.com/glycerine/turnpike.(*Realm).getPeer
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:41 +0x31f

goroutine 9 [select]:
github.com/glycerine/turnpike.(*Realm).handleSession(0xc82015c3f0, 0x1d290c8, 0xc820144df0, 0xa14e9af8b268e, 0xc820015140, 0xc82001e8a0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/realm.go:99 +0x255a
github.com/glycerine/turnpike.(*defaultRouter).Accept.func1(0xc82015c3f0, 0x1d290c8, 0xc820144df0, 0xa14e9af8b268e, 0xc820015140, 0xc82001e8a0, 0xc820092aa0, 0xc820081be0)
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:162 +0x77
created by github.com/glycerine/turnpike.(*defaultRouter).Accept
    /Users/jaten/go/src/github.com/glycerine/turnpike/router.go:167 +0x16d2

goroutine 36 [chan receive]:
github.com/glycerine/turnpike.(*Client).Receive(0xc820092b40)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:212 +0xcc
created by github.com/glycerine/turnpike.(*Client).JoinRealm
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:99 +0x90e

goroutine 37 [chan receive]:
github.com/glycerine/turnpike.(*Client).Receive(0xc820092b90)
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:212 +0xcc
created by github.com/glycerine/turnpike.(*Client).JoinRealm
    /Users/jaten/go/src/github.com/glycerine/turnpike/client.go:99 +0x90e
exit status 2
FAIL    github.com/glycerine/turnpike   1.681s
~/go/src/github.com/glycerine/turnpike$
jcelliott commented 9 years ago

I recently ran the data race tests as well. It's on the top of my to-do list for turnpike. Or if you would like to take a stab at it I would gladly accept a pull request.

glycerine commented 9 years ago

Hi Joshua,

I looked for a quick fix, but reasoning about the existing code is somewhat difficult.

Forgive me -- I hope this doesn't sound too harsh. There's tons of good stuff in your code. You're 90% of the way there. It's just that the goroutine organization is hard to follow.

If you don't mind a few gentle suggestions, here are a couple of best-practices that I learned about goroutine work from writing a distributed queuing system in Go (https://github.com/glycerine/goq).

a) Keep a 1:1 relationship between each go-routine and its private state that is represented in its own struct. That is, each go-routine that is spawned should have its own instance of a struct that captures its private state.

Given this struct that simply holds the private state for that go-routine, no other goroutine should ever touch the struct of anther. This avoids all data races. The struct members can and should all be private, except for the control channels such as RequestStop and Done which should be public, as they represent the control surface of the goroutine: the interaction points between go-routines are always sends and receives on channels.

b) think of each independent goroutine as a communicating state machine, and use the go func() { for { select { case <- channel1: ; case <- channel2: ... } } } idiom. This is clean. It makes it easy to reason about what is going on. It ensures that even if you pass ownership of pointers around on channels, there are never any races. I elaborate on this in my general notes to new Go programmers here https://github.com/glycerine/thinkgo.

c) shutdown of goroutines is the hardest part of getting your system right. It doesn't look like this has been addressed at the moment. The RequestStop channel described above allows you to ask a goroutine to stop. Just before the goroutine returns (and hence stops), it closes its Done channel. This is the same control pattern that Gustavo Niemeyer's tomb implements, but I find the name tomb just a tad too morbid for my taste. Nonetheless it is a critically important idea: clean shutdown is what enables you to test the system in lots of different test scenarios. net.Context (i.e. https://blog.golang.org/context) is another attempt at standardizing shutdown semantics. I find it over engineered, but at least it represents an attempt to standardize goroutine shutdown, which is helpful in communicating ideas between engineers.

My conclusion after reading through client.go and related files is that quite a bit of refactoring is order in order to bring the code into alignment with the above principals. I'm not sure I have the time immediately to do this myself, although I would like to as wamp seems like a really nice design point in the space of "things-that-bridge-browsers and backend systems". And I also didn't feel comfortable undertaking big changes without discussing such changes with you, the original author. Let me know your thoughts.

Jason

glycerine commented 8 years ago

bump. What are your thoughts on a big refactor, @jcelliott?

mattbaird commented 8 years ago

@jcelliott @glycerine - what if someone wrote a one pager on the refactor and we commitment and worked on it as a team? I know for me, doing a big refactor without knowing if the PR will be accepted is a major deterrent (as @glycerine says in his last sentence).

glycerine commented 8 years ago

@mattbaird Thanks for commenting. I may be able to allocate some time to this. @marshauf, what are your current thoughts? If Joshua/@jcelliott is swamped, perhaps a friendly fork would be better.

jcelliott commented 8 years ago

I have been swamped the last couple months, but I should have some time now. I was planning on doing some work on turnpike this week. I could definitely use some help going forward with fixing these concurrency issues.

@mattbaird what did you have in mind for a "one pager"?

mattbaird commented 8 years ago

@jcelliott more like guidance on the scope of changes required to follow the practices @glycerine outlined. Maybe one or two code examples based on the turnpike code base. I'm not the biggest fan of prescriptive requirements docs, however if we want to parcel out this work to get the disruption over asap, sharing the concepts more concretely might help accelerate.

yanfali commented 8 years ago

I have a few fixes for the most obvious race conditions exposed by -race in a branch, which at least point out which data structures are problematic. Feel free to take a look at them and use this as the basis for your own implementation.

Hopefully, you guys can apply similar fixes to v2 and make this awesome project even better. Apologies if someone has already been working on this, but I needed something right away for a prototype.

https://github.com/awakenetworks/turnpike/tree/race-fixes

mourad commented 8 years ago

I believe all concurrency issues are now being handled by PR #119.

I would like to hear back from those involved before closing this issue.