Panic with `inconsistent target for`

intosKai commented 6 months ago

I found the problem, that on linux/amd64 RAFT randomly throws panic

Dragonboat version

v3.3.4

Expected behavior

No error

Actual behavior

transport: inconsistent target for [00000:28902], 146.190.25.190:61785:0.0.0.0:61785
panic: inconsistent target for [00000:28902], 146.190.25.190:61785:0.0.0.0:61785

goroutine 254 [running]:
github.com/lni/goutils/logutil/capnslog.(*PackageLogger).Panicf(0xc000013c68, {0x154baee?, 0xc000058ef0?}, {0xc0011f6360?, 0xc0000f1f50?, 0xc0010fb7f0?})
    /root/go/pkg/mod/github.com/lni/goutils@v1.3.0/logutil/capnslog/pkg_logger.go:88 +0xb5
github.com/lni/dragonboat/v3/logger.(*capnsLog).Panicf(0xc0000f1f50?, {0x154baee?, 0x411745?}, {0xc0011f6360?, 0x12c6300?, 0xc0010e2001?})
    /root/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.4/logger/capnslogger.go:74 +0x25
github.com/lni/dragonboat/v3/logger.(*dragonboatLogger).Panicf(0xc0010c4022?, {0x154baee, 0x21}, {0xc0011f6360, 0x3, 0x3})
    /root/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.4/logger/logger.go:132 +0x51
github.com/lni/dragonboat/v3/internal/transport.(*Registry).Add(0xc000377e80, 0x0, 0xab5599ff3e5c4ae6, {0xc0010c4022, 0xd})
    /root/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.4/internal/transport/registry.go:85 +0x3ab
github.com/lni/dragonboat/v3/internal/transport.(*Transport).handleRequest(0xc000168008, {{0xc0002c4000, 0x1, 0x1}, 0x1, {0xc0010c4022, 0xd}, 0xd2})
    /root/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.4/internal/transport/transport.go:324 +0x32d
github.com/lni/dragonboat/v3/internal/transport.(*TCP).serveConn(0xc0000e1688, {0x17ea1d0, 0xc0000586c0})
    /root/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.4/internal/transport/tcp.go:559 +0x25b
github.com/lni/dragonboat/v3/internal/transport.(*TCP).Start.func2.3()
    /root/go/pkg/mod/github.com/lni/dragonboat/v3@v3.3.4/internal/transport/tcp.go:488 +0x28
github.com/lni/goutils/syncutil.(*Stopper).runWorker.func1()
    /root/go/pkg/mod/github.com/lni/goutils@v1.3.0/syncutil/stopper.go:79 +0x123
created by github.com/lni/goutils/syncutil.(*Stopper).runWorker in goroutine 226
    /root/go/pkg/mod/github.com/lni/goutils@v1.3.0/syncutil/stopper.go:74 +0xc6

kevburnsjr commented 6 months ago

That's a pretty weird error log. [00000:28902] suggests replica number 28,902 of shard 0. Is this a real world use case or some sort of chaos test?

intosKai commented 6 months ago

It really happens. Moreover everything is okay when I trying to run same configuration on my mac, instead remote server. It is possible that something wrong with VDS network configuration?

intosKai commented 6 months ago

upgrade to v3.3.5 solved the issue

lni commented 6 months ago

@intosKai

Hi, I don't think there is anything changed between v3.3.4 and v3.3.5 in that regard.

Based on the error log, it seems to me that the issue is trigger when replica number 28,902 of shard 0 had its address somehow changed during execution. Any chance that it is caused by network configurations?

lni / dragonboat