segmentio / ecs-logs

Log forwarder for services ran by ecs-agent.
MIT License
114 stars 16 forks source link

Panic on concurrent map writes #65

Closed calvinfo closed 6 years ago

calvinfo commented 6 years ago

We're seeing the following error on some busier hosts

fatal error: concurrent map writes
goroutine 121938 [running]:
runtime.throw(0x8cc74c, 0x15)
        /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc4216f1e10 sp=0xc4216f1df0
runtime.mapdelete(0x8438e0, 0xc420017f50, 0xc4216f1ee8)
        /usr/local/go/src/runtime/hashmap.go:612 +0x322 fp=0xc4216f1e78 sp=0xc4216f1e10
github.com/segmentio/ecs-logs/lib/cloudwatchlogs.(*client).Open(0xc4200b31a0, 0xc42121162e, 0x1d, 0xc4233ddf14, 0x24, 0xce21e0, 0xc42508dc80, 0x7f65a85106b0, 0xc424b4fe90)
        /go/src/github.com/segmentio/ecs-logs/lib/cloudwatchlogs/client.go:50 +0x29c fp=0xc4216f1f08 sp=0xc4216f1e78
main.write(0xcdfce0, 0xc4200b31a0, 0x7ffeaf489e0e, 0xe, 0xc42121162e, 0x1d, 0xc4233ddf14, 0x24, 0xc420d1aa80, 0x2, ...)
        /go/src/github.com/segmentio/ecs-logs/main.go:318 +0x99 fp=0xc4216f1f80 sp=0xc4216f1f08
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc4216f1f88 sp=0xc4216f1f80
created by main.flush
        /go/src/github.com/segmentio/ecs-logs/main.go:357 +0x61b
goroutine 1 [select]:
main.main()
        /go/src/github.com/segmentio/ecs-logs/main.go:136 +0x1586
goroutine 17 [syscall, 46 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1
goroutine 5 [syscall, 46 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()