vasya4k / gopcep

Implementation of PCEP and a TE Controller written in Go
MIT License
15 stars 4 forks source link

fatal error: sync: Unlock of unlocked RWMutex #3

Open paddy01 opened 6 months ago

paddy01 commented 6 months ago

After a bit of hassle I managed to get gopcep to function. Have to control it from remote so had to hackaround in your .js and stuff to get it functioning. But after finally getting everything up and running I attached it to a Juniper box and configured all knobs that has to be configured.. when it finally started to send bgp-ls updates gopcep broke down with the error "fatal error: sync: Unlock of unlocked RWMutex".

Complete log from start to crash:

time="2024-03-01T11:16:28.011505629+01:00" level=info msg="running with config" config="&{{192.0.29.134 12345 [Efqwdoin9148]} {192.168.68.191 1443 /opt/gopcep/certs/gopcep.crt /opt/gopcep/certs/gopcep.key pcep1 pcep false} {192.0.29.134 4189 30} {true false 2006-01-02T15:04:05.999999999Z07:00 true 6} {65001 192.0.29.134}}" event="red config" topic=config
time="2024-03-01T11:16:28.013969407+01:00" level=info msg="Add a peer configuration for:192.0.28.1" Topic=Peer
time="2024-03-01T11:16:28.014157763+01:00" level=debug msg="IdleHoldTimer expired" Duration=0 Key=192.0.28.1 Topic=Peer
time="2024-03-01T11:16:28.014239674+01:00" level=debug msg="state changed" Key=192.0.28.1 Topic=Peer new=BGP_FSM_ACTIVE old=BGP_FSM_IDLE reason=idle-hold-timer-expired
time="2024-03-01T11:16:33.014572683+01:00" level=debug msg="try to connect" Key=192.0.28.1 Topic=Peer
time="2024-03-01T11:16:33.015577716+01:00" level=debug msg="state changed" Key=192.0.28.1 Topic=Peer new=BGP_FSM_OPENSENT old=BGP_FSM_ACTIVE reason=new-connection
time="2024-03-01T11:16:33.017454706+01:00" level=debug msg="state changed" Key=192.0.28.1 Topic=Peer new=BGP_FSM_OPENCONFIRM old=BGP_FSM_OPENSENT reason=open-msg-received
time="2024-03-01T11:16:33.023334817+01:00" level=info msg="Peer Up" Key=192.0.28.1 State=BGP_FSM_OPENCONFIRM Topic=Peer
time="2024-03-01T11:16:33.023406285+01:00" level=debug msg="state changed" Key=192.0.28.1 Topic=Peer new=BGP_FSM_ESTABLISHED old=BGP_FSM_OPENCONFIRM reason=open-msg-negotiated
time="2024-03-01T11:16:33.023521812+01:00" level=info msg="added new peer" event=new_peer peer_addr=192.0.28.1 peer_as=65536 peer_state=ESTABLISHED type=bgp
time="2024-03-01T11:16:33.023825584+01:00" level=debug msg="received update" Key=192.0.28.1 Topic=Peer attributes="[{Origin: i} 65536 {LsAttributes: {Node Flags: ********} {Node Name: ROUTER-1} {ISIS Area ID: [73 6 102]} {Local RouterID IPv4: 192.0.28.2} } {MpReach(ls): {Nexthop: 192.0.28.1, NLRIs: [NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8002 ISIS-L2:0 } }]}}]" nlri="[]" withdrawals="[]"
time="2024-03-01T11:16:33.02396796+01:00" level=debug msg="create Destination" Nlri="NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8002 ISIS-L2:0 } }" Topic=Table
time="2024-03-01T11:16:33.024030844+01:00" level=debug msg="create Destination" Nlri="NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8002 ISIS-L2:0 } }" Topic=Table
time="2024-03-01T11:16:33.024060652+01:00" level=debug msg="computeKnownBestPath knownPathList: 1" Topic=Table
time="2024-03-01T11:16:33.024127992+01:00" level=debug msg="From me, ignore." Data="{ NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8002 ISIS-L2:0 } } | src: { 192.0.28.1 | as: 65536, id: 192.0.28.1 }, nh: 192.0.28.1 }" Key=192.0.28.1 Topic=Peer
time="2024-03-01T11:16:33.024216845+01:00" level=debug msg="received update" Key=192.0.28.1 Topic=Peer attributes="[{Origin: i} 65536 {LsAttributes: {Node Flags: ********} {Node Name: ROUTER-2} {ISIS Area ID: [73 6 102]} {Local RouterID IPv4: 192.0.28.7} } {MpReach(ls): {Nexthop: 192.0.28.1, NLRIs: [NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8007 ISIS-L2:0 } }]}}]" nlri="[]" withdrawals="[]"
time="2024-03-01T11:16:33.024328085+01:00" level=debug msg="create Destination" Nlri="NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8007 ISIS-L2:0 } }" Topic=Table
time="2024-03-01T11:16:33.024364319+01:00" level=debug msg="create Destination" Nlri="NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8007 ISIS-L2:0 } }" Topic=Table
time="2024-03-01T11:16:33.024422387+01:00" level=debug msg="computeKnownBestPath knownPathList: 1" Topic=Table
time="2024-03-01T11:16:33.025514283+01:00" level=debug msg="From me, ignore." Data="{ NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8007 ISIS-L2:0 } } | src: { 192.0.28.1 | as: 65536, id: 192.0.28.1 }, nh: 192.0.28.1 }" Key=192.0.28.1 Topic=Peer
time="2024-03-01T11:16:33.025655046+01:00" level=debug msg="received update" Key=192.0.28.1 Topic=Peer attributes="[{Origin: i} 65536 {LsAttributes: } {MpReach(ls): {Nexthop: 192.0.28.1, NLRIs: [NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:1::/127] } }]}}]" nlri="[]" withdrawals="[]"
time="2024-03-01T11:16:33.025826554+01:00" level=debug msg="create Destination" Nlri="NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:1::/127] } }" Topic=Table
time="2024-03-01T11:16:33.025876012+01:00" level=debug msg="create Destination" Nlri="NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:1::/127] } }" Topic=Table
time="2024-03-01T11:16:33.025941534+01:00" level=debug msg="computeKnownBestPath knownPathList: 1" Topic=Table
time="2024-03-01T11:16:33.025964768+01:00" level=debug msg="From me, ignore." Data="{ NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:1::/127] } } | src: { 192.0.28.1 | as: 65536, id: 192.0.28.1 }, nh: 192.0.28.1 }" Key=192.0.28.1 Topic=Peer
time="2024-03-01T11:16:33.026052874+01:00" level=debug msg="received update" Key=192.0.28.1 Topic=Peer attributes="[{Origin: i} 65536 {LsAttributes: {Node Flags: ********} {Node Name: ROUTER-3} {ISIS Area ID: [73 6 102]} {Local RouterID IPv4: 192.0.28.1} {SR Capabilities: Flags:192 SRGB Ranges: 100000:249999 } {SR Algorithms: [0 1]} } {MpReach(ls): {Nexthop: 192.0.28.1, NLRIs: [NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8001 ISIS-L2:0 } }]}}]" nlri="[]" withdrawals="[]"
time="2024-03-01T11:16:33.026142839+01:00" level=debug msg="create Destination" Nlri="NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8001 ISIS-L2:0 } }" Topic=Table
time="2024-03-01T11:16:33.026231538+01:00" level=debug msg="create Destination" Nlri="NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8001 ISIS-L2:0 } }" Topic=Table
time="2024-03-01T11:16:33.026257558+01:00" level=debug msg="computeKnownBestPath knownPathList: 1" Topic=Table
time="2024-03-01T11:16:33.026305949+01:00" level=debug msg="From me, ignore." Data="{ NLRI { NODE { AS:65536 BGP-LS ID:0 1920.002.8001 ISIS-L2:0 } } | src: { 192.0.28.1 | as: 65536, id: 192.0.28.1 }, nh: 192.0.28.1 }" Key=192.0.28.1 Topic=Peer
time="2024-03-01T11:16:33.026345769+01:00" level=debug msg="received update" Key=192.0.28.1 Topic=Peer attributes="[{Origin: i} 65536 {LsAttributes: } {MpReach(ls): {Nexthop: 192.0.28.1, NLRIs: [NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:7::/127] } }]}}]" nlri="[]" withdrawals="[]"
time="2024-03-01T11:16:33.026425135+01:00" level=debug msg="create Destination" Nlri="NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:7::/127] } }" Topic=Table
time="2024-03-01T11:16:33.026526463+01:00" level=debug msg="create Destination" Nlri="NLRI { PREFIXv6 { LOCAL_NODE: 1920.002.8001 PREFIX: [fc00:0:1:7::/127] } }" Topic=Table
time="2024-03-01T11:16:33.026615125+01:00" level=debug msg="computeKnownBestPath knownPathList: 1" Topic=Table
fatal error: sync: Unlock of unlocked RWMutex
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xab4b7d]

goroutine 18 [running]:
sync.fatal({0xdf1b0d?, 0xc00014ee00?})
        /usr/local/go/src/runtime/panic.go:1007 +0x18
sync.(*RWMutex).Unlock(0xc0001248d0)
        /usr/local/go/src/sync/rwmutex.go:208 +0x45
panic({0xcb0a20?, 0x2709170?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
gopcep/controller.(*TopoView).HandleNodeNLRI(0xc000070c60, 0xc000250140, 0xc0004f8000)
        /opt/gopcep/controller/bgpls.go:167 +0x23d
gopcep/controller.(*TopoView).Monitor(0xc000070c60, 0xc0004f8000)
        /opt/gopcep/controller/bgpls.go:247 +0x2e5
gopcep/controller.(*Controller).StartBGPLS.func2(0xc000225f88?)
        /opt/gopcep/controller/bgpls.go:522 +0x25
github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable.func2()
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3827 +0x309
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable in goroutine 7
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3789 +0xd4

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7fc288fabdb8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x8?, 0x278f1a0?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000333580)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000333580)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00006e3c0)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc00006e3c0)
        /usr/local/go/src/net/tcpsock.go:327 +0x30
gopcep/pcep.ListenForNewSession({0x2138fb0, 0xc00016c000}, 0xc00012a1a0)
        /opt/gopcep/pcep/session.go:567 +0x29a
main.startController(0xc000366e40?)
        /opt/gopcep/main.go:142 +0x735
github.com/urfave/cli/v2.(*App).RunContext(0xc000374680, {0x213dc18, 0x278f1a0}, {0xc000034190, 0x1, 0x1})
        /root/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:322 +0x928
github.com/urfave/cli/v2.(*App).Run(...)
        /root/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224
main.main()
        /opt/gopcep/main.go:166 +0x165

goroutine 7 [chan receive]:
gopcep/controller.(*Controller).StartBGPLS(0xc00016c000)
        /opt/gopcep/controller/bgpls.go:531 +0x92b
created by gopcep/controller.Start in goroutine 1
        /opt/gopcep/controller/controller.go:424 +0x538

goroutine 8 [select]:
gopcep/controller.Start.func1()
        /opt/gopcep/controller/controller.go:428 +0x98
created by gopcep/controller.Start in goroutine 1
        /opt/gopcep/controller/controller.go:426 +0x576

goroutine 9 [IO wait]:
internal/poll.runtime_pollWait(0x7fc288fabeb0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x7?, 0x1?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000333300)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000333300)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00032dba0)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc00032dba0)
        /usr/local/go/src/net/tcpsock.go:327 +0x30
google.golang.org/grpc.(*Server).Serve(0xc0003ff180, {0x213c580, 0xc00032dba0})
        /root/go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:739 +0x363
gopcep/grpcapi.Start.func1()
        /opt/gopcep/grpcapi/api.go:144 +0x45
created by gopcep/grpcapi.Start in goroutine 1
        /opt/gopcep/grpcapi/api.go:143 +0x9a5

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0x7fc288fabcc0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x9?, 0xc00018cd30?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000333600)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000333600)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00006e3e0)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc00006e3e0)
        /usr/local/go/src/net/tcpsock.go:327 +0x30
crypto/tls.(*listener).Accept(0xc0000321c8)
        /usr/local/go/src/crypto/tls/tls.go:66 +0x27
net/http.(*Server).Serve(0xc0004d23c0, {0x213c7f0, 0xc0000321c8})
        /usr/local/go/src/net/http/server.go:3255 +0x33e
net/http.(*Server).ServeTLS(0xc0004d23c0, {0x213c580, 0xc00006e3e0}, {0xc000308240, 0x1c}, {0xc000308260, 0x1c})
        /usr/local/go/src/net/http/server.go:3325 +0x486
net/http.(*Server).ListenAndServeTLS(0xc0004d23c0, {0xc000308240, 0x1c}, {0xc000308260, 0x1c})
        /usr/local/go/src/net/http/server.go:3482 +0x125
gopcep/restapi.Start.func3()
        /opt/gopcep/restapi/restapi.go:211 +0x46
created by gopcep/restapi.Start in goroutine 1
        /opt/gopcep/restapi/restapi.go:210 +0x526

goroutine 11 [chan send]:
github.com/osrg/gobgp/pkg/server.(*watcher).notify(...)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4110
github.com/osrg/gobgp/pkg/server.(*BgpServer).notifyWatcher(...)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4146
github.com/osrg/gobgp/pkg/server.(*BgpServer).notifyBestWatcher(0xc0000ddb08, {0xc000483430, 0x1, 0xc000483418?}, {0xc000032d98, 0x0, 0xdca2e0?})
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:763 +0x435
github.com/osrg/gobgp/pkg/server.(*BgpServer).propagateUpdateToNeighbors(0xc0000ddb08, 0xc000338aa0, 0xc000339770, {0xc000483418, 0x1, 0x1}, 0x1)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:1245 +0x10f
github.com/osrg/gobgp/pkg/server.(*BgpServer).propagateUpdate(0xc0000ddb08, 0xc000338aa0, {0xc000483408, 0x1, 0xc000338aa0?})
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:1216 +0x665
github.com/osrg/gobgp/pkg/server.(*BgpServer).handleFSMMessage(0xc0000ddb08, 0xc000338aa0, 0xc0000ca600)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:1607 +0x77b
github.com/osrg/gobgp/pkg/server.(*BgpServer).Serve.func1(0xc0000ca600)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:394 +0x5b8
github.com/osrg/gobgp/pkg/server.(*BgpServer).Serve(0xc0000ddb08)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:434 +0x48a
created by gopcep/controller.(*Controller).StartBGPLS in goroutine 7
        /opt/gopcep/controller/bgpls.go:477 +0xa5

goroutine 12 [select]:
github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer.func1()
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3849 +0x22e
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer in goroutine 7
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3843 +0xbe

goroutine 33 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc00006eaa0)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 14 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc00006e760)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 34 [select]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).established(0xc00012f650, {0x213de80, 0xc0003392c0})
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1757 +0x454
github.com/osrg/gobgp/pkg/server.(*fsmHandler).loop(0xc00012f650, {0x213de80, 0xc0003392c0}, 0xc0001fa670?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1870 +0x236
created by github.com/osrg/gobgp/pkg/server.newFSMHandler in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:406 +0x1e5

goroutine 16 [runnable]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc00006e7a0)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 17 [chan receive]:
github.com/osrg/gobgp/pkg/server.(*watcher).loop(0xc000338b90)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4114 +0x5b
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable.func1.(*BgpServer).watch.2 in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4299 +0x70f

goroutine 19 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc00006e7c0)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 20 [chan receive]:
github.com/osrg/gobgp/pkg/server.(*watcher).loop(0xc000338be0)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4114 +0x5b
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer.func1.(*BgpServer).watch.3 in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4299 +0x70f

goroutine 35 [select]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).sendMessageloop(0xc00012f650, {0x213de80, 0xc0003392c0}, 0xc00006ea40?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1683 +0x1ae
created by github.com/osrg/gobgp/pkg/server.(*fsmHandler).established in goroutine 34
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1741 +0x1aa

goroutine 36 [IO wait]:
internal/poll.runtime_pollWait(0x7fc288fabbc8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc000333e80?, 0xc000125848?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000333e80, {0xc000125848, 0x13, 0x13})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc000333e80, {0xc000125848?, 0x1?, 0xc17088a04184e03f?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000482ff8, {0xc000125848?, 0x18?, 0x272f0a0?})
        /usr/local/go/src/net/net.go:179 +0x45
io.ReadAtLeast({0x2134620, 0xc000482ff8}, {0xc000125848, 0x13, 0x13}, 0x13)
        /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
github.com/osrg/gobgp/pkg/server.readAll({0x2143fa8, 0xc000482ff8}, 0x13)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:796 +0x6d
github.com/osrg/gobgp/pkg/server.(*fsmHandler).recvMessageWithError(0xc00012f650)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:933 +0x3a
github.com/osrg/gobgp/pkg/server.(*fsmHandler).recvMessageloop(0xc00012f650, {0x213de80?, 0xc0003392c0?}, 0xc0001fa670?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1721 +0x5b
created by github.com/osrg/gobgp/pkg/server.(*fsmHandler).established in goroutine 34
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1743 +0x257
vasya4k commented 6 months ago

I did not know someone was still interested in this project. Saying this I have been working on this project recently to revive the front end and also look at how I can make mutex locking better. I was thinking about making it useful for DWDW as PCEP is used there as well. I will try to take a look at this issue today or tomorrow as I have everything set up.

vasya4k commented 6 months ago

I moved the mutex lock so that part should work now. But that also means you would probably get an error in the logs somewhere.

err := ptypes.UnmarshalAny(lsMessage, &NLRINode) if err != nil { logrus.Println(err) return } node := &Node{ ASN: NLRINode.LocalNode.Asn, IGPRouteID: NLRINode.LocalNode.IgpRouterId, } var LsAttribute api.LsAttribute for _, item := range p.Pattrs { if ptypes.Is(item, &LsAttribute) { err := ptypes.UnmarshalAny(item, &LsAttribute) if err != nil { logrus.Println(err) return } node.RouterID = LsAttribute.Node.LocalRouterId node.Name = LsAttribute.Node.Name node.SRRangeStart = int(LsAttribute.Node.SrCapabilities.Ranges[0].Begin) node.SRRangeEnd = int(LsAttribute.Node.SrCapabilities.Ranges[0].End) } }

Because UnmarshalAny does not work for some reason. Potentially due to new attributes in BGP LS. I can fix it too if you are interested and can provide some data on Juniper version etc...

paddy01 commented 6 months ago

I moved the mutex lock so that part should work now. But that also means you would probably get an error in the logs somewhere.

Yep now it crashes with

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xab4b20]

goroutine 18 [running]:
gopcep/controller.(*TopoView).HandleNodeNLRI(0xc000070c00, 0xc0002c9270, 0xc0004f7110)
        /opt/gopcep/controller/bgpls.go:165 +0x1e0
gopcep/controller.(*TopoView).Monitor(0xc000070c00, 0xc0004f7110)
        /opt/gopcep/controller/bgpls.go:246 +0x2e5
gopcep/controller.(*Controller).StartBGPLS.func2(0xc000225f88?)
        /opt/gopcep/controller/bgpls.go:521 +0x25
github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable.func2()
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3827 +0x309
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable in goroutine 7
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3789 +0xd4

I can fix it too if you are interested and can provide some data on Juniper version etc...

I don't mind being guinea pig if you like - I'm actually looking at ODL at the same time.. running JunOS 21.4R3 atm.

vasya4k commented 6 months ago

I added a nil pointer check so it will should not crash anymore. I also added a debug check to print out whole LsAttribute as json.

paddy01 commented 6 months ago

I added a nil pointer check so it will should not crash anymore. I also added a debug check to print out whole LsAttribute as json.

Seems to have done the trick, it's no longer crashing at least :)

paddy01 commented 5 months ago

This issue is back! :( I established a bgp-ls session with a ios-xr device and when it started actually sending data the crash came back :(

fatal error: sync: Unlock of unlocked RWMutex

goroutine 21 [running]:
sync.fatal({0xdf77e3?, 0xc000176540?})
        /usr/local/go/src/runtime/panic.go:1007 +0x18
sync.(*RWMutex).Unlock(0xc0002b4918)
        /usr/local/go/src/sync/rwmutex.go:208 +0x45
panic({0xd79b60?, 0xc0006190b0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
gopcep/controller.(*TopoView).HandleLinkNLRI(0xc000072c60, 0xc0003c1770, 0xc0003a2270)
        /opt/gopcep/controller/bgpls.go:205 +0x3d9
gopcep/controller.(*TopoView).Monitor(0xc000072c60, 0xc0003a2270)
        /opt/gopcep/controller/bgpls.go:252 +0x2c5
gopcep/controller.(*Controller).StartBGPLS.func2(0xc00023ff88?)
        /opt/gopcep/controller/bgpls.go:525 +0x25
github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable.func2()
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3827 +0x309
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable in goroutine 7
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3789 +0xd4

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7fe2e3b32db8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x8?, 0x0?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000353600)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000353600)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000070520)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc000070520)
        /usr/local/go/src/net/tcpsock.go:327 +0x30
gopcep/pcep.ListenForNewSession({0x213eeb0, 0xc000154090}, 0xc0001441a0)
        /opt/gopcep/pcep/session.go:567 +0x29a
main.startController(0xc00037ce40?)
        /opt/gopcep/main.go:142 +0x735
github.com/urfave/cli/v2.(*App).RunContext(0xc00038c4e0, {0x2143b18, 0x279e120}, {0xc000034190, 0x1, 0x1})
        /root/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:322 +0x928
github.com/urfave/cli/v2.(*App).Run(...)
        /root/go/pkg/mod/github.com/urfave/cli/v2@v2.3.0/app.go:224
main.main()
        /opt/gopcep/main.go:166 +0x165

goroutine 7 [chan receive]:
gopcep/controller.(*Controller).StartBGPLS(0xc000154090)
        /opt/gopcep/controller/bgpls.go:534 +0x92b
created by gopcep/controller.Start in goroutine 1
        /opt/gopcep/controller/controller.go:428 +0x538

goroutine 8 [select]:
gopcep/controller.Start.func1()
        /opt/gopcep/controller/controller.go:432 +0x98
created by gopcep/controller.Start in goroutine 1
        /opt/gopcep/controller/controller.go:430 +0x576

goroutine 9 [IO wait]:
internal/poll.runtime_pollWait(0x7fe2e3b32eb0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x7?, 0x1?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000353380)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000353380)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000349cc0)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc000349cc0)
        /usr/local/go/src/net/tcpsock.go:327 +0x30
google.golang.org/grpc.(*Server).Serve(0xc0003ff340, {0x2142480, 0xc000349cc0})
        /root/go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go:739 +0x363
gopcep/grpcapi.Start.func1()
        /opt/gopcep/grpcapi/api.go:144 +0x45
created by gopcep/grpcapi.Start in goroutine 1
        /opt/gopcep/grpcapi/api.go:143 +0x9a5

goroutine 10 [IO wait]:
internal/poll.runtime_pollWait(0x7fe2e3b32cc0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x9?, 0xc0001a71d0?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000353680)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc000353680)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000070540)
        /usr/local/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc000070540)
        /usr/local/go/src/net/tcpsock.go:327 +0x30
crypto/tls.(*listener).Accept(0xc0000e4240)
        /usr/local/go/src/crypto/tls/tls.go:66 +0x27
net/http.(*Server).Serve(0xc0004ee3c0, {0x21426f0, 0xc0000e4240})
        /usr/local/go/src/net/http/server.go:3255 +0x33e
net/http.(*Server).ServeTLS(0xc0004ee3c0, {0x2142480, 0xc000070540}, {0xc00032a2c0, 0x1c}, {0xc00032a2e0, 0x1c})
        /usr/local/go/src/net/http/server.go:3325 +0x486
net/http.(*Server).ListenAndServeTLS(0xc0004ee3c0, {0xc00032a2c0, 0x1c}, {0xc00032a2e0, 0x1c})
        /usr/local/go/src/net/http/server.go:3482 +0x125
gopcep/restapi.Start.func3()
        /opt/gopcep/restapi/restapi.go:211 +0x46
created by gopcep/restapi.Start in goroutine 1
        /opt/gopcep/restapi/restapi.go:210 +0x526

goroutine 11 [select]:
reflect.rselect({0xc000208fa0, 0x5, 0xc000687da8?})
        /usr/local/go/src/runtime/select.go:589 +0x2c5
reflect.Select({0xc000421440?, 0x5, 0x0?})
        /usr/local/go/src/reflect/value.go:3169 +0x5ca
github.com/osrg/gobgp/pkg/server.(*BgpServer).Serve(0xc0002b8488)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:418 +0x427
created by gopcep/controller.(*Controller).StartBGPLS in goroutine 7
        /opt/gopcep/controller/bgpls.go:480 +0xa5

goroutine 12 [select]:
github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer.func1()
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3849 +0x22e
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer in goroutine 7
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:3843 +0xbe

goroutine 52 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc000071000)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 14 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc000070960)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 53 [select]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).established(0xc00014a230, {0x2143d80, 0xc00035de50})
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1757 +0x454
github.com/osrg/gobgp/pkg/server.(*fsmHandler).loop(0xc00014a230, {0x2143d80, 0xc00035de50}, 0xc000210be0?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1870 +0x236
created by github.com/osrg/gobgp/pkg/server.newFSMHandler in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:406 +0x1e5

goroutine 48 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc000070f80)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 17 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc000070a00)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 49 [select]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).established(0xc000149e30, {0x2143d80, 0xc00035dc20})
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1757 +0x454
github.com/osrg/gobgp/pkg/server.(*fsmHandler).loop(0xc000149e30, {0x2143d80, 0xc00035dc20}, 0xc000210be0?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1870 +0x236
created by github.com/osrg/gobgp/pkg/server.newFSMHandler in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:406 +0x1e5

goroutine 19 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc000070a40)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 20 [chan receive]:
github.com/osrg/gobgp/pkg/server.(*watcher).loop(0xc00035ceb0)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4114 +0x5b
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorTable.func1.(*BgpServer).watch.2 in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4299 +0x70f

goroutine 22 [select]:
github.com/eapache/channels.(*InfiniteChannel).infiniteBuffer(0xc000070a60)
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:49 +0xe5
created by github.com/eapache/channels.NewInfiniteChannel in goroutine 11
        /root/go/pkg/mod/github.com/eapache/channels@v1.1.0/infinite_channel.go:19 +0x156

goroutine 23 [chan receive]:
github.com/osrg/gobgp/pkg/server.(*watcher).loop(0xc00035cf50)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4114 +0x5b
created by github.com/osrg/gobgp/pkg/server.(*BgpServer).MonitorPeer.func1.(*BgpServer).watch.3 in goroutine 11
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/server.go:4299 +0x70f

goroutine 54 [select]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).sendMessageloop(0xc00014a230, {0x2143d80, 0xc00035de50}, 0xc000070de0?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1683 +0x1ae
created by github.com/osrg/gobgp/pkg/server.(*fsmHandler).established in goroutine 53
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1741 +0x1aa

goroutine 50 [select]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).sendMessageloop(0xc000149e30, {0x2143d80, 0xc00035dc20}, 0xc000070f40?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1683 +0x1ae
created by github.com/osrg/gobgp/pkg/server.(*fsmHandler).established in goroutine 49
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1741 +0x1aa

goroutine 51 [runnable]:
github.com/osrg/gobgp/pkg/server.(*fsmHandler).recvMessageloop(0xc000149e30, {0x2143d80?, 0xc00035dc20?}, 0xc000210be0?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1723 +0x94
created by github.com/osrg/gobgp/pkg/server.(*fsmHandler).established in goroutine 49
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1743 +0x257

goroutine 55 [IO wait]:
internal/poll.runtime_pollWait(0x7fe2e3b32ad0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc0000cc080?, 0xc0003a87f8?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0000cc080, {0xc0003a87f8, 0x13, 0x13})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0000cc080, {0xc0003a87f8?, 0x0?, 0xc173cb5c77c04693?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00049d148, {0xc0003a87f8?, 0x18?, 0x273e020?})
        /usr/local/go/src/net/net.go:179 +0x45
io.ReadAtLeast({0x213a520, 0xc00049d148}, {0xc0003a87f8, 0x13, 0x13}, 0x13)
        /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
github.com/osrg/gobgp/pkg/server.readAll({0x2149ea8, 0xc00049d148}, 0x13)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:796 +0x6d
github.com/osrg/gobgp/pkg/server.(*fsmHandler).recvMessageWithError(0xc00014a230)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:933 +0x3a
github.com/osrg/gobgp/pkg/server.(*fsmHandler).recvMessageloop(0xc00014a230, {0x2143d80?, 0xc00035de50?}, 0xc000210be0?)
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1721 +0x5b
created by github.com/osrg/gobgp/pkg/server.(*fsmHandler).established in goroutine 53
        /root/go/pkg/mod/github.com/osrg/gobgp@v0.0.0-20210402043138-915bfc2d8189/pkg/server/fsm.go:1743 +0x257

goroutine 56 [IO wait]:
internal/poll.runtime_pollWait(0x7fe2e3b329d8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc00040dd00?, 0xc00009f400?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00040dd00, {0xc00009f400, 0x400, 0x400})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00040dd00, {0xc00009f400?, 0x400?, 0x273e020?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00049d8a8, {0xc00009f400?, 0xc4fea0?, 0x1?})
        /usr/local/go/src/net/net.go:179 +0x45
gopcep/pcep.startPCEPSession({0x2149ea8, 0xc00049d8a8}, {0x213eeb0, 0xc000154090})
        /opt/gopcep/pcep/session.go:524 +0x4da
created by gopcep/pcep.ListenForNewSession in goroutine 1
        /opt/gopcep/pcep/session.go:588 +0x626

goroutine 57 [sleep]:
time.Sleep(0x6fc23ac00)
        /usr/local/go/src/runtime/time.go:195 +0x115
gopcep/pcep.(*Session).StartKeepAlive(0xc00002ac60)
        /opt/gopcep/pcep/session.go:293 +0x435
created by gopcep/pcep.(*Session).HandleNewMsg in goroutine 56
        /opt/gopcep/pcep/session.go:388 +0x1c5

goroutine 58 [select]:
gopcep/pcep.(*Session).HandleDeadTimer(0xc00002ac60)
        /opt/gopcep/pcep/session.go:319 +0xad
created by gopcep/pcep.(*Session).HandleNewMsg in goroutine 56
        /opt/gopcep/pcep/session.go:389 +0x207

goroutine 59 [select]:
gopcep/controller.(*Controller).watchSession(0xc000154090, 0xc00002ac60)
        /opt/gopcep/controller/controller.go:460 +0x85
created by gopcep/controller.Start.func1 in goroutine 8
        /opt/gopcep/controller/controller.go:439 +0x474

goroutine 60 [IO wait]:
internal/poll.runtime_pollWait(0x7fe2e3b328e0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc00040df00?, 0xc000282400?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00040df00, {0xc000282400, 0x400, 0x400})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00040df00, {0xc000282400?, 0x400?, 0xc000065008?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00049d8b8, {0xc000282400?, 0xc4fea0?, 0x1?})
        /usr/local/go/src/net/net.go:179 +0x45
gopcep/pcep.startPCEPSession({0x2149ea8, 0xc00049d8b8}, {0x213eeb0, 0xc000154090})
        /opt/gopcep/pcep/session.go:524 +0x4da
created by gopcep/pcep.ListenForNewSession in goroutine 1
        /opt/gopcep/pcep/session.go:588 +0x626

goroutine 61 [sleep]:
time.Sleep(0x6fc23ac00)
        /usr/local/go/src/runtime/time.go:195 +0x115
gopcep/pcep.(*Session).StartKeepAlive(0xc00002b050)
        /opt/gopcep/pcep/session.go:293 +0x435
created by gopcep/pcep.(*Session).HandleNewMsg in goroutine 60
        /opt/gopcep/pcep/session.go:388 +0x1c5

goroutine 62 [select]:
gopcep/pcep.(*Session).HandleDeadTimer(0xc00002b050)
        /opt/gopcep/pcep/session.go:319 +0xad
created by gopcep/pcep.(*Session).HandleNewMsg in goroutine 60
        /opt/gopcep/pcep/session.go:389 +0x207

goroutine 63 [select]:
gopcep/controller.(*Controller).watchSession(0xc000154090, 0xc00002b050)
        /opt/gopcep/controller/controller.go:460 +0x85
created by gopcep/controller.Start.func1 in goroutine 8
        /opt/gopcep/controller/controller.go:439 +0x474