lovoo / goka

Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.
BSD 3-Clause "New" or "Revised" License
2.35k stars 175 forks source link

EmitSync stuck in channel #417

Closed rockwithamoon closed 1 year ago

rockwithamoon commented 1 year ago

Hello.

I've noticed on rare occasions that EmitSync() does not return. From the stack trace it seems that done channel is not closing from https://github.com/lovoo/goka/blob/v1.1.1/emitter.go#L114

goroutine 6730623 [chan receive, 41 minutes]:
github.com/lovoo/goka.(*Emitter).EmitSyncWithHeaders(0xc08e5da290, {0xc042657368, 0x14}, {0x102b720, 0xc05475da40}, 0x1299ce0)
        /go/src/bitbucket.org/xx/xx/vendor/github.com/lovoo/goka/emitter.go:114 +0x131
github.com/lovoo/goka.(*Emitter).EmitSync(...)
        /go/src/bitbucket.org/xx/xx/vendor/github.com/lovoo/goka/emitter.go:120
bitbucket.org/xx/xx/core/ke.(*Kafka).mainLoop.func1({{0xc042657368, 0xc05ecc4fd0}, 0xc05475da40, 0xc070ce5500})
        /go/src/bitbucket.org/xx/xx/core/ke/ke.go:74 +0x4b
created by bitbucket.org/xx/xx/core/ke.(*Kafka).mainLoop
        /go/src/bitbucket.org/xx/xx/core/ke/ke.go:73 +0x7d

It's an older version of goka, I'll upgrade to latest but this happens once a month or so it will take a while for feedback.

        github.com/lovoo/goka v1.1.1
        github.com/Shopify/sarama v1.31.0 // indirect
frairon commented 1 year ago

Hmm, that looks really strange. The logic inside the emitter is not that complicated. It's merely wrapping sarama. Does the error occur during shutdown of the emitter or at normal runtime?

rockwithamoon commented 1 year ago

This is during normal runtime. I'm not 100% sure yet if this happens during Kafka re-partitioning. I've updated to goka v1.1.7 and sarama v1.38.1. If it occurs again I'll try to debug more as I've seen some receiving channels in waiting in sarama/async_producer.go.

rockwithamoon commented 1 year ago

After upgrading to sarama v1.38.1 this behaviour isn't reproduced. I'm closing this. Thanks.