Data race detected in internal/utilization/utilization.go

cyrilc-pro commented 3 weeks ago

After upgrading to v3.34.0, a data race condition is detected in my tests.

Description

After upgrading from Go 1.22 to Go 1.23 and from NR GoAgent 3.32.0 to 3.34.0, my tests are failing because of the following race condition:

WARNING: DATA RACE
Write at 0x00c002452b10 by goroutine 12799:
  github.com/newrelic/go-agent/v3/internal/utilization.gatherAWS()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/aws.go:39 +0xa7
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2.1()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:159 +0xdc

Previous read at 0x00c002452b10 by goroutine 12800:
  github.com/newrelic/go-agent/v3/internal/utilization.(*vendors).AnySet()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:87 +0x9d
  github.com/newrelic/go-agent/v3/internal/utilization.gatherAzure()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/azure.go:33 +0x43
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2.1()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:159 +0xdc

Goroutine 12799 (running) created at:
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:152 +0x207
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:168 +0x3fa
  github.com/newrelic/go-agent/v3/internal/utilization.Gather()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:119 +0x264
  github.com/newrelic/go-agent/v3/newrelic.config.createConnectJSON()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/config.go:1065 +0x1ad
  github.com/newrelic/go-agent/v3/newrelic.connectAttempt()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/collector.go:349 +0x64e
  github.com/newrelic/go-agent/v3/newrelic.(*app).connectRoutine()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/internal_app.go:141 +0x164
  github.com/newrelic/go-agent/v3/newrelic.newApp.gowrap2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/internal_app.go:465 +0x33

Goroutine 12800 (finished) created at:
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:152 +0x207
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:172 +0x431
  github.com/newrelic/go-agent/v3/internal/utilization.Gather()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/internal/utilization/utilization.go:119 +0x264
  github.com/newrelic/go-agent/v3/newrelic.config.createConnectJSON()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/config.go:1065 +0x1ad
  github.com/newrelic/go-agent/v3/newrelic.connectAttempt()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/collector.go:349 +0x64e
  github.com/newrelic/go-agent/v3/newrelic.(*app).connectRoutine()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/internal_app.go:141 +0x164
  github.com/newrelic/go-agent/v3/newrelic.newApp.gowrap2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/v3@v3.34.0/newrelic/internal_app.go:465 +0x33

Steps to Reproduce

Tests are run from inside a Docker container.

Expected Behavior

No race condition.

NR Diag results

N/A

Your Environment

N/A

Reproduction case

N/A

Additional context

N/A

nr-swilloughby commented 1 week ago

To ensure that we can verify that the fix we release for this when it's been diagnosed covers your specific case can you provide a testcase which reproduces the problem you are reporting here, in a way that doesn't expose anything proprietary or relies on external components not needed to verify this condition? With a race condition it can be important to know we're trying the same set of call flows that you are when the race condition is occurring. Thanks.

nr-swilloughby commented 1 week ago

Although I haven't been able to reproduce your error, analysis of the code base has identified a potential source for a race condition that could be triggered in some circumstances. I'm putting in a correction for this issue and we'll see if this resolves the problem you're having.

cyrilc-pro commented 6 days ago

Thanks. It's hard to provide a simple test case, as the issue was detected on an integration test for a full application.

nr-swilloughby commented 5 days ago

I have merged a proposed fix to our develop branch which is where our code gets staged shortly before release. Could you try pulling a copy of the agent from that branch and try compiling with that to see if that solves your issue?

newrelic / go-agent