loggie-io / loggie

A lightweight, cloud-native data transfer agent and aggregator
https://loggie-io.github.io/docs-en/
Apache License 2.0
1.26k stars 168 forks source link

使用grpc source时,sink invoke loop有概率卡住 #534

Open hmlv opened 1 year ago

hmlv commented 1 year ago

What version of Loggie?

v1.4.0

Expected Behavior

Actual Behavior

https://github.com/loggie-io/loggie/blob/main/pkg/source/grpc/batch.go#L149 程序运行一段时间后卡在batchChain的ack函数中,bc.ackEvents无法写入

Steps to Reproduce the Problem

grpc source + dev sink,queue使用channel 将queue的batchBytes设置为1,让event单个发到sink,上游持续发压几分钟后出现sink不消费; 怀疑grpc source的batch.go实现有问题,在run函数中要消费bc.batchChan和bc.ackEvents两个channel,bc.batchChan的消费最后会讲数据写入queue的in channel,bc.ackEvents的消费会处理queue的out channel,而queue内部的loop自身in channel的消费需要写入自身的out channel,见https://github.com/loggie-io/loggie/blob/main/pkg/queue/channel/queue.go#L143,两个loop形成了循环依赖,在消费不及时时就会出现grpc source batch loop卡住(需要消耗queue的out,但是loop又被queue的in卡住)

ethfoo commented 1 year ago

这个应该是和这个https://github.com/loggie-io/loggie/issues/228 相同的问题,会尽快修复

hmlv commented 1 year ago

不是的,228那个是event_pool卡住,我测试这个case是,event_pool能正常get

machine3 commented 1 year ago

batchChan卡住会走不到ackChan,ackChan无法消费会导致batchChan一直卡着