search

apache / dubbo-go

Go Implementation For Apache Dubbo .
https://dubbo.apache.org/
Apache License 2.0
4.73k stars 930 forks source link

发现有Data Race现象，会导致程序挂掉 #2277

Open youfu-fun opened 1 year ago

youfu-fun commented 1 year ago

What happened:
早上发现某个pod重启了很多次，查看日志发现是因为 fatal error: concurrent map writes

What you expected to happen: 正常运行，不会导致panic。

How to reproduce it (as minimally and precisely as possible): 运行一个consumer， 在运行时加入-race运行检测

Anything else we need to know? 使用的dubbo-go版本 线上： 3.0.5 本地： 3.0.5 和 master

pod日志信息： image

本地完整日志信息：

==================
WARNING: DATA RACE
Write at 0x00c000940928 by goroutine 140:
  dubbo.apache.org/dubbo-go/v3/registry/directory.(*RegistryDirectory).setNewInvokers()
      /Users/youfu/Documents/dubbo-go/registry/directory/directory.go:240 +0x90
  dubbo.apache.org/dubbo-go/v3/registry/directory.(*RegistryDirectory).refreshAllInvokers()
      /Users/youfu/Documents/dubbo-go/registry/directory/directory.go:206 +0x1b8
  dubbo.apache.org/dubbo-go/v3/registry/directory.(*RegistryDirectory).NotifyAll.func1()
      /Users/youfu/Documents/dubbo-go/registry/directory/directory.go:123 +0x64

Previous read at 0x00c000940928 by main goroutine:
  dubbo.apache.org/dubbo-go/v3/registry/directory.(*RegistryDirectory).IsAvailable()
      /Users/youfu/Documents/dubbo-go/registry/directory/directory.go:428 +0x3c
  dubbo.apache.org/dubbo-go/v3/cluster/cluster/base.(*BaseClusterInvoker).IsAvailable()
      /Users/youfu/Documents/dubbo-go/cluster/cluster/base/cluster_invoker.go:68 +0x8c
  dubbo.apache.org/dubbo-go/v3/cluster/cluster/failover.(*failoverClusterInvoker).IsAvailable()
      <autogenerated>:1 +0x34
  dubbo.apache.org/dubbo-go/v3/config.(*ConsumerConfig).Load()
      /Users/youfu/Documents/dubbo-go/config/consumer_config.go:154 +0x8d4
  dubbo.apache.org/dubbo-go/v3/config.(*RootConfig).Start.func1()
      /Users/youfu/Documents/dubbo-go/config/root_config.go:212 +0x44
  sync.(*Once).doSlow()
      /Users/youfu/Documents/go/go1.19.3/src/sync/once.go:74 +0xb0
  sync.(*Once).Do()
      /Users/youfu/Documents/go/go1.19.3/src/sync/once.go:65 +0x40
  dubbo.apache.org/dubbo-go/v3/config.(*RootConfig).Start()
      /Users/youfu/Documents/dubbo-go/config/root_config.go:210 +0x48
  dubbo.apache.org/dubbo-go/v3/config.(*RootConfig).Init()
      /Users/youfu/Documents/dubbo-go/config/root_config.go:205 +0x744
  dubbo.apache.org/dubbo-go/v3/config.Load()
      /Users/youfu/Documents/dubbo-go/config/config_loader.go:59 +0x160
  main.InitService()
      /Users/youfu/Documents/demo/grpc.go:14 +0xc0
  main.main()
      /Users/youfu/Documents/demo/main.go:59 +0x2c0

Goroutine 140 (running) created at:
  dubbo.apache.org/dubbo-go/v3/registry/directory.(*RegistryDirectory).NotifyAll()
      /Users/youfu/Documents/dubbo-go/registry/directory/directory.go:123 +0x120
  dubbo.apache.org/dubbo-go/v3/registry/servicediscovery.(*ServiceInstancesChangedListenerImpl).OnEvent()
      /Users/youfu/Documents/dubbo-go/registry/servicediscovery/service_instances_changed_listener_impl.go:142 +0x800
  dubbo.apache.org/dubbo-go/v3/registry/nacos.(*nacosServiceDiscovery).AddListener.func1()
      /Users/youfu/Documents/dubbo-go/registry/nacos/service_discovery.go:278 +0x63c
  github.com/nacos-group/nacos-sdk-go/clients/naming_client.(*SubscribeCallback).ServiceChanged()
      /Users/youfu/Documents/go/pkg/mod/github.com/nacos-group/nacos-sdk-go@v1.1.4/clients/naming_client/subscribe_callback.go:94 +0x18c
  github.com/nacos-group/nacos-sdk-go/clients/naming_client.(*NamingClient).Subscribe()
      /Users/youfu/Documents/go/pkg/mod/github.com/nacos-group/nacos-sdk-go@v1.1.4/clients/naming_client/naming_client.go:339 +0x274
  dubbo.apache.org/dubbo-go/v3/registry/nacos.(*nacosServiceDiscovery).AddListener()
      /Users/youfu/Documents/dubbo-go/registry/nacos/service_discovery.go:246 +0x428
  dubbo.apache.org/dubbo-go/v3/registry/servicediscovery.(*ServiceDiscoveryRegistry).SubscribeURL()
      /Users/youfu/Documents/dubbo-go/registry/servicediscovery/service_discovery_registry.go:247 +0x39c
  dubbo.apache.org/dubbo-go/v3/registry/servicediscovery.(*ServiceMappingChangedListenerImpl).OnEvent()
      /Users/youfu/Documents/dubbo-go/registry/servicediscovery/service_mapping_change_listener_impl.go:84 +0x2d0
  dubbo.apache.org/dubbo-go/v3/registry/servicediscovery.(*ServiceDiscoveryRegistry).Subscribe()
      /Users/youfu/Documents/dubbo-go/registry/servicediscovery/service_discovery_registry.go:216 +0x454
  dubbo.apache.org/dubbo-go/v3/registry/directory.(*RegistryDirectory).Subscribe()
      /Users/youfu/Documents/dubbo-go/registry/directory/directory.go:107 +0x114
  dubbo.apache.org/dubbo-go/v3/registry/protocol.(*registryProtocol).Refer()
      /Users/youfu/Documents/dubbo-go/registry/protocol/protocol.go:164 +0x3b0
  dubbo.apache.org/dubbo-go/v3/config.(*ReferenceConfig).Refer()
      /Users/youfu/Documents/dubbo-go/config/reference_config.go:238 +0xb40
  dubbo.apache.org/dubbo-go/v3/config.(*ConsumerConfig).Load()
      /Users/youfu/Documents/dubbo-go/config/consumer_config.go:132 +0x3c0
  dubbo.apache.org/dubbo-go/v3/config.(*RootConfig).Start.func1()
      /Users/youfu/Documents/dubbo-go/config/root_config.go:212 +0x44
  sync.(*Once).doSlow()
      /Users/youfu/Documents/go/go1.19.3/src/sync/once.go:74 +0xb0
  sync.(*Once).Do()
      /Users/youfu/Documents/go/go1.19.3/src/sync/once.go:65 +0x40
  dubbo.apache.org/dubbo-go/v3/config.(*RootConfig).Start()
      /Users/youfu/Documents/dubbo-go/config/root_config.go:210 +0x48
  dubbo.apache.org/dubbo-go/v3/config.(*RootConfig).Init()
      /Users/youfu/Documents/dubbo-go/config/root_config.go:205 +0x744
  dubbo.apache.org/dubbo-go/v3/config.Load()
      /Users/youfu/Documents/dubbo-go/config/config_loader.go:59 +0x160
  main.InitService()
      /Users/youfu/Documents/demo/grpc.go:14 +0xc0
  main.main()
      /Users/youfu/Documents/demo/main.go:59 +0x2c0
==================
AlexStocks commented 1 year ago

@georgehao pls check this issue. maybe we have fixed it.