search

DataDog / datadog-lambda-extension

Datadog Lambda Extension
Apache License 2.0
81 stars 5 forks source link

Version 65 of extension panics due to concurrent map modification #463

Open kostaspap opened 2 days ago

kostaspap commented 2 days ago

Hello,

we noticed the following issue when running Version 65 of the extension. Error seems to point to an issue with concurrent map modifications being performed here.

Panic 

/usr/local/go/src/net/http/transport.go:1544 +0x118
created by net/http.(*Transport).startDialConnForLocked in goroutine 16
/usr/local/go/src/net/http/transport.go:1544
net/http.(*Transport).startDialConnForLocked.func1()
goroutine 50 [runnable]:
/tmp/dd/datadog-agent/cmd/serverless/main.go:121 +0x24c
created by main.runAgent in goroutine 1
/tmp/dd/datadog-agent/cmd/serverless/main.go:300 +0x1b4
"main.startTelemetryCollection(0x0?, {0x4000110000, 0x24}, 0x4000396070, 0x400016c1e0, {0x1c21f50, 0x29d8b40})"
/tmp/dd/datadog-agent/pkg/serverless/registration/telemetry_api.go:42 +0x100
"github.com/DataDog/datadog-agent/pkg/serverless/registration.EnableTelemetryCollection({{0x4000110000, 0x24}, {0x400010b8f0, 0x2a}, 0x12a05f200, {0x0, 0x0}, 0x1fbc, {0x1953359, 0xc}, ...})"
/tmp/dd/datadog-agent/pkg/serverless/registration/telemetry_api.go:57 +0x184
"github.com/DataDog/datadog-agent/pkg/serverless/registration.subscribeTelemetry({0x4000110000, 0x24}, {0x400010b8f0, 0x2a}, 0x12a05f200, {0x1bf4ea0?, 0x400007d9e0?})"
/tmp/dd/datadog-agent/pkg/serverless/registration/telemetry_api.go:110
github.com/DataDog/datadog-agent/pkg/serverless/registration.sendLogRegistrationRequest(...)
/usr/local/go/src/net/http/client.go:590
net/http.(*Client).Do(...)
/usr/local/go/src/net/http/client.go:725 +0x6a8
"net/http.(*Client).do(0x4000670f30, 0x40003e88c0)"
/usr/local/go/src/net/http/client.go:180 +0x9c
"net/http.(*Client).send(0x4000670f30, 0x40003e88c0, {0x2ce9b0?, 0x524848?, 0x299f500?})"
/usr/local/go/src/net/http/client.go:259 +0x4ac
"net/http.send(0x40003e88c0, {0x1bf4cc0, 0x40001c4000}, {0x6f65f4?, 0x8?, 0x299f500?})"
/usr/local/go/src/net/http/roundtrip.go:30 +0x1c
"net/http.(*Transport).RoundTrip(0x40003e8a00?, 0x1bf4cc0?)"
/usr/local/go/src/net/http/transport.go:633 +0x8f4
"net/http.(*Transport).roundTrip(0x40001c4000, 0x40003e8a00)"
/usr/local/go/src/net/http/transport.go:1468 +0x364
"net/http.(*Transport).getConn(0x40001c4000, 0x4000133270, {{}, 0x0, {0x400010b8f0, 0x4}, {0x40004c5f10, 0xe}, 0x0})"
goroutine 16 [select]:
/tmp/dd/datadog-agent/comp/dogstatsd/server/server.go:527 +0x300
created by github.com/DataDog/datadog-agent/comp/dogstatsd/server.(*server).handleMessages in goroutine 1
/tmp/dd/datadog-agent/comp/dogstatsd/server/server_worker.go:58 +0x8c
github.com/DataDog/datadog-agent/comp/dogstatsd/server.(*worker).run(0x40001d5280)
goroutine 35 [select]:
/tmp/dd/datadog-agent/comp/dogstatsd/server/server.go:527 +0x300
created by github.com/DataDog/datadog-agent/comp/dogstatsd/server.(*server).handleMessages in goroutine 1
/tmp/dd/datadog-agent/comp/dogstatsd/server/server_worker.go:58 +0x8c
github.com/DataDog/datadog-agent/comp/dogstatsd/server.(*worker).run(0x40001d5240)
goroutine 34 [select]:
/tmp/dd/datadog-agent/pkg/util/containers/metrics/provider/registry.go:88 +0x104
created by github.com/DataDog/datadog-agent/pkg/util/containers/metrics/provider.(*collectorRegistry).run.func1 in goroutine 1
/tmp/dd/datadog-agent/pkg/util/containers/metrics/provider/registry.go:96 +0xbc
"github.com/DataDog/datadog-agent/pkg/util/containers/metrics/provider.(*collectorRegistry).collectorDiscovery(0x4000310c60, {0x1c0e5c8, 0x29d8b40}, 0x40001d51c0, {{0x0?, 0x0?}, 0x0?})"
goroutine 33 [select]:
/tmp/dd/datadog-agent/comp/dogstatsd/listeners/udp.go:113 +0x70
created by github.com/DataDog/datadog-agent/comp/dogstatsd/listeners.(*UDPListener).Listen in goroutine 1
/tmp/dd/datadog-agent/comp/dogstatsd/listeners/udp.go:115 +0x50
github.com/DataDog/datadog-agent/comp/dogstatsd/listeners.(*UDPListener).Listen.func1()
/tmp/dd/datadog-agent/comp/dogstatsd/listeners/udp.go:123 +0x140
github.com/DataDog/datadog-agent/comp/dogstatsd/listeners.(*UDPListener).listen(0x40001df2c0)
/usr/local/go/src/net/udpsock.go:158 +0x4c
"net.(*UDPConn).ReadFrom(0x40000dad18, {0x400057c000, 0x2000, 0x2000})"
/usr/local/go/src/net/udpsock.go:149 +0x2c
"net.(*UDPConn).readFromUDP(0x40000dad18, {0x400057c000?, 0x1?, 0x1?}, 0x8964e0?)"
/usr/local/go/src/net/udpsock_posix.go:52 +0x13c
"net.(*UDPConn).readFrom(0x40002a7410?, {0x400057c000?, 0x0?, 0x40002a7410?}, 0x40002a7410)"
/usr/local/go/src/net/fd_posix.go:66 +0x28
"net.(*netFD).readFromInet4(0x4000414800, {0x400057c000?, 0x40004e25e8?, 0x478f0c?}, 0xffff561cc4d8?)"
/usr/local/go/src/internal/poll/fd_unix.go:248 +0x1e8
"internal/poll.(*FD).ReadFromInet4(0x4000414800, {0x400057c000, 0x2000, 0x2000}, 0x40004e25e0)"
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
"internal/poll.(*pollDesc).wait(0x4000414800?, 0x400057c000?, 0x0)"
/usr/local/go/src/runtime/netpoll.go:351 +0xa0
"internal/poll.runtime_pollWait(0xffff572274e8, 0x72)"
goroutine 32 [IO wait]:
/tmp/dd/datadog-agent/pkg/status/health/health.go:63 +0xb8
created by github.com/DataDog/datadog-agent/pkg/status/health.(*catalog).register in goroutine 1
/tmp/dd/datadog-agent/pkg/status/health/health.go:91 +0x50
github.com/DataDog/datadog-agent/pkg/status/health.(*catalog).run(0x299e6e0)
goroutine 31 [chan receive]:
/tmp/dd/datadog-agent/comp/dogstatsd/packets/assembler.go:42 +0x11c
created by github.com/DataDog/datadog-agent/comp/dogstatsd/packets.NewAssembler in goroutine 1
/tmp/dd/datadog-agent/comp/dogstatsd/packets/assembler.go:48 +0x5c
github.com/DataDog/datadog-agent/comp/dogstatsd/packets.(*Assembler).flushLoop(0x40001d5180)
goroutine 30 [select]:
/tmp/dd/datadog-agent/comp/dogstatsd/packets/buffer.go:37 +0x140
created by github.com/DataDog/datadog-agent/comp/dogstatsd/packets.NewBuffer in goroutine 1
/tmp/dd/datadog-agent/comp/dogstatsd/packets/buffer.go:43 +0x5c
github.com/DataDog/datadog-agent/comp/dogstatsd/packets.(*Buffer).flushLoop(0x40001df1a0)
goroutine 29 [select]:
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:40 +0xc8
created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:63 +0x38
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0x400002e630)
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:50 +0x88
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0x400002e630)
/usr/local/go/src/sync/cond.go:71 +0xcc
sync.(*Cond).Wait(0x40001d4fc0)
/usr/local/go/src/runtime/sema.go:587 +0x154
"sync.runtime_notifyListWait(0x40001d4fd0, 0x0)"
goroutine 26 [sync.Cond.Wait]:
/tmp/dd/datadog-agent/pkg/aggregator/demultiplexer_serverless.go:70 +0x584
created by github.com/DataDog/datadog-agent/pkg/aggregator.InitAndStartServerlessDemultiplexer in goroutine 1
/tmp/dd/datadog-agent/pkg/aggregator/demultiplexer_serverless.go:86 +0x124
github.com/DataDog/datadog-agent/pkg/aggregator.(*ServerlessDemultiplexer).Run(0x40001de6c0)
/tmp/dd/datadog-agent/pkg/aggregator/time_sampler_worker.go:83 +0x144
github.com/DataDog/datadog-agent/pkg/aggregator.(*timeSamplerWorker).run(0x40000b4820)
goroutine 25 [select]:
/usr/local/go/src/net/http/transport.go:1875 +0x1098
created by net/http.(*Transport).dialConn in goroutine 10
/usr/local/go/src/net/http/transport.go:2519 +0x9c
net/http.(*persistConn).writeLoop(0x4000252b40)
goroutine 13 [select]:
/usr/local/go/src/net/http/transport.go:1874 +0x1050
created by net/http.(*Transport).dialConn in goroutine 10
/usr/local/go/src/net/http/transport.go:2205 +0x138
net/http.(*persistConn).readLoop(0x4000252b40)
/usr/local/go/src/bufio/bufio.go:148 +0x60
"bufio.(*Reader).Peek(0x400007d020, 0x1)"
/usr/local/go/src/bufio/bufio.go:110 +0xf8
bufio.(*Reader).fill(0x400007d020)
/usr/local/go/src/net/http/transport.go:2052 +0x50
"net/http.(*persistConn).Read(0x4000252b40, {0x4000330000?, 0x16b0d80?, 0x40002a6ea0?})"
/usr/local/go/src/net/net.go:189 +0x34
"net.(*conn).Read(0x4000074d08, {0x4000330000?, 0x4000088cb8?, 0x41e00c?})"
/usr/local/go/src/net/fd_posix.go:55 +0x28
"net.(*netFD).Read(0x40004da580, {0x4000330000?, 0x1bf4860?, 0x1615a40?})"
/usr/local/go/src/internal/poll/fd_unix.go:165 +0x1fc
"internal/poll.(*FD).Read(0x40004da580, {0x4000330000, 0x1000, 0x1000})"
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
"internal/poll.(*pollDesc).wait(0x40004da580?, 0x4000330000?, 0x0)"
/usr/local/go/src/runtime/netpoll.go:351 +0xa0
"internal/poll.runtime_pollWait(0xffff572275f0, 0x72)"
goroutine 12 [IO wait]:
/tmp/dd/datadog-agent/cmd/serverless/main.go:119 +0x15c
created by main.runAgent in goroutine 1
/tmp/dd/datadog-agent/cmd/serverless/main.go:338 +0xd0
"main.startTraceAgent(0x0?, 0x40003960e0, 0xb9394f06c8196fa, 0x400016c1e0)"
/tmp/dd/datadog-agent/pkg/serverless/trace/trace.go:101 +0x94
"github.com/DataDog/datadog-agent/pkg/serverless/trace.StartServerlessTraceAgent(0x50?, {0x1bf47a0, 0x4000375a10}, 0x40003960e0, 0xb9394f06c8196fa)"
/tmp/dd/datadog-agent/pkg/serverless/trace/trace.go:84 +0x2c
github.com/DataDog/datadog-agent/pkg/serverless/trace.(*LoadConfig).Load(0x4000375a10)
/tmp/dd/datadog-agent/comp/core/config/config.go:78 +0x394
"github.com/DataDog/datadog-agent/comp/core/config.NewServerlessConfig({0x19666ca, 0x16})"
/tmp/dd/datadog-agent/comp/core/config/config.go:91 +0x5c
"github.com/DataDog/datadog-agent/comp/core/config.newConfig({{{}}, {{0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0}, {0x19503fc, 0xa}, ...}, ...})"
/tmp/dd/datadog-agent/comp/core/config/setup.go:60 +0x230
"github.com/DataDog/datadog-agent/comp/core/config.setupConfig({0x1c30500, 0x400014b950}, {0x1bff470, 0x4000159680})"
/tmp/dd/datadog-agent/pkg/config/setup/config.go:1718
github.com/DataDog/datadog-agent/pkg/config/setup.LoadWithoutSecret(...)
/tmp/dd/datadog-agent/pkg/config/setup/config.go:1924 +0x1b0
"github.com/DataDog/datadog-agent/pkg/config/setup.LoadDatadogCustom({0x1c30500, 0x400014b950}, {0x1953449, 0xc}, {{0x0, 0x0}, 0x0}, {0x29d8b40, 0x0, 0x0})"
/tmp/dd/datadog-agent/pkg/config/setup/config.go:1660 +0xe4
"github.com/DataDog/datadog-agent/pkg/config/setup.LoadProxyFromEnv({0x1c30500, 0x400014b950})"
/tmp/dd/datadog-agent/pkg/config/model/viper.go:565 +0x100
"github.com/DataDog/datadog-agent/pkg/config/model.(*safeConfig).UnmarshalKey(0x400014b950, {0x194861a, 0x5}, {0x15b97a0, 0x40001d5300}, {0x0, 0x0, 0x0})"
/root/go/pkg/mod/github.com/!data!dog/viper@v1.13.5/viper.go:871 +0x70
"github.com/DataDog/viper.(*Viper).UnmarshalKey(0x40004210e0, {0x194861a?, 0x5?}, {0x15b97a0, 0x40001d5300}, {0x0, 0x0, 0x0})"
/root/go/pkg/mod/github.com/!data!dog/viper@v1.13.5/viper.go:1869
github.com/DataDog/viper.(*Viper).AllSettings(...)
/root/go/pkg/mod/github.com/!data!dog/viper@v1.13.5/viper.go:1898 +0x274
"github.com/DataDog/viper.(*Viper).allSettings(0x40004210e0, 0x400046f898)"
goroutine 14 [runnable]:
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:40 +0xc8
created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:63 +0x38
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0x40004d8000)
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:50 +0x88
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0x40004d8000)
/usr/local/go/src/sync/cond.go:71 +0xcc
sync.(*Cond).Wait(0x40004d28c0)
/usr/local/go/src/runtime/sema.go:587 +0x154
"sync.runtime_notifyListWait(0x40004d28d0, 0x0)"
goroutine 24 [sync.Cond.Wait]:
/tmp/dd/datadog-agent/pkg/serverless/daemon/daemon.go:142 +0x4e0
created by github.com/DataDog/datadog-agent/pkg/serverless/daemon.StartDaemon in goroutine 1
/tmp/dd/datadog-agent/pkg/serverless/daemon/daemon.go:143 +0x24
github.com/DataDog/datadog-agent/pkg/serverless/daemon.StartDaemon.func1()
/usr/local/go/src/net/http/server.go:3259 +0x84
net/http.(*Server).ListenAndServe(0x400016c2d0)
/usr/local/go/src/net/http/server.go:3330 +0x294
"net/http.(*Server).Serve(0x400016c2d0, {0x1c08828, 0x40001d4140})"
/usr/local/go/src/net/tcpsock.go:372 +0x2c
net.(*TCPListener).Accept(0x40001d4140)
/usr/local/go/src/net/tcpsock_posix.go:159 +0x28
net.(*TCPListener).accept(0x40001d4140)
/usr/local/go/src/net/fd_unix.go:172 +0x28
net.(*netFD).accept(0x4000414000)
/usr/local/go/src/internal/poll/fd_unix.go:620 +0x24c
internal/poll.(*FD).Accept(0x4000414000)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
"internal/poll.(*pollDesc).wait(0x4000414000?, 0x41fee4?, 0x0)"
/usr/local/go/src/runtime/netpoll.go:351 +0xa0
"internal/poll.runtime_pollWait(0xffff572276f8, 0x72)"
goroutine 9 [IO wait]:
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:40 +0xc8
created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:63 +0x38
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0x400014b4d0)
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:50 +0x88
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0x400014b4d0)
/usr/local/go/src/sync/cond.go:71 +0xcc
sync.(*Cond).Wait(0x400040cb40)
/usr/local/go/src/runtime/sema.go:587 +0x154
"sync.runtime_notifyListWait(0x400040cb50, 0x0)"
goroutine 5 [sync.Cond.Wait]:
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:40 +0xc8
created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:63 +0x38
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0x400014b3b0)
/root/go/pkg/mod/github.com/cihub/seelog@v0.0.0-20151216151435-d2c6e5aa9fbf/behavior_asynclooplogger.go:50 +0x88
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0x400014b3b0)
/usr/local/go/src/sync/cond.go:71 +0xcc
sync.(*Cond).Wait(0x400040c940)
/usr/local/go/src/runtime/sema.go:587 +0x154
"sync.runtime_notifyListWait(0x400040c950, 0x0)"
goroutine 4 [sync.Cond.Wait]:
/root/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1099 +0xe8
created by github.com/patrickmn/go-cache.runJanitor in goroutine 1
/root/go/pkg/mod/github.com/patrickmn/go-cache@v2.1.0+incompatible/cache.go:1079 +0x78
"github.com/patrickmn/go-cache.(*janitor).Run(0x40001ca470, 0x40001d4100)"
goroutine 3 [select]:
/tmp/dd/datadog-agent/cmd/serverless/main.go:74 +0xb8
main.main()
/tmp/dd/datadog-agent/pkg/util/fxutil/oneshot.go:54 +0x258
"github.com/DataDog/datadog-agent/pkg/util/fxutil.OneShot({0x1615b40?, 0x1a34530?}, {0x40003c0370, 0x1, 0x1})"
/tmp/dd/datadog-agent/pkg/util/fxutil/args.go:72 +0x94
github.com/DataDog/datadog-agent/pkg/util/fxutil.(*delayedFxInvocation).call(0x4000202600?)
/usr/local/go/src/reflect/value.go:365 +0x94
"reflect.Value.Call({0x1615b40?, 0x1a34530?, 0x29d8b40?}, {0x40000dd4d0?, 0x0?, 0x931660?})"
/usr/local/go/src/reflect/value.go:581 +0x97c
"reflect.Value.call({0x1615b40?, 0x1a34530?, 0x400048fcb8?}, {0x1947ac6, 0x4}, {0x40000dd4d0, 0x1, 0x1bf50a0?})"
/tmp/dd/datadog-agent/cmd/serverless/main.go:126 +0x260
"main.runAgent({0x1c21f50, 0x29d8b40})"
/usr/local/go/src/sync/waitgroup.go:118 +0x74
sync.(*WaitGroup).Wait(0x40004c5c80)
/usr/local/go/src/runtime/sema.go:71 +0x2c
sync.runtime_Semacquire(0x40000021c0?)
goroutine 1 [semacquire]:
/tmp/dd/datadog-agent/cmd/serverless/main.go:120 +0x1bc
created by main.runAgent in goroutine 1
/tmp/dd/datadog-agent/cmd/serverless/main.go:326 +0x4c
"main.startOtlpAgent(0x0?, 0x40001d4300, 0x400016c1e0)"
/tmp/dd/datadog-agent/pkg/serverless/otlp/otlp.go:64 +0x58
github.com/DataDog/datadog-agent/pkg/serverless/otlp.IsEnabled()
/tmp/dd/datadog-agent/comp/otelcol/otlp/config.go:113
github.com/DataDog/datadog-agent/comp/otelcol/otlp.IsEnabled(...)
/tmp/dd/datadog-agent/comp/otelcol/otlp/config.go:127 +0x38
"github.com/DataDog/datadog-agent/comp/otelcol/otlp.hasSection({0x1c2cf30?, 0x400014b950?}, {0x194d0be, 0x8})"
/tmp/dd/datadog-agent/comp/otelcol/otlp/config.go:62 +0xc4
"github.com/DataDog/datadog-agent/comp/otelcol/otlp.readConfigSection({0x1c2cf30, 0x400014b950}, {0x19524d3, 0xb})"
/tmp/dd/datadog-agent/pkg/config/model/viper.go:286 +0xac
github.com/DataDog/datadog-agent/pkg/config/model.(*safeConfig).AllKeysLowercased(0x16adb60?)
/root/go/pkg/mod/github.com/!data!dog/viper@v1.13.5/viper.go:1792 +0x1a0
github.com/DataDog/viper.(*Viper).AllKeys(0x40004210e0)
/root/go/pkg/mod/github.com/!data!dog/viper@v1.13.5/viper.go:1839 +0x1e8
"github.com/DataDog/viper.(*Viper).flattenAndMergeMap(0x40004210e0, 0x50bb88?, 0x40004556b0, {0x0, 0x0})"
/root/go/pkg/mod/github.com/!data!dog/viper@v1.13.5/viper.go:1826 +0x100
"github.com/DataDog/viper.(*Viper).flattenAndMergeMap(0x40004210e0, 0x4000475d18?, 0x40004c86f0, {0x195035c, 0x5})"
goroutine 15 [running]:
fatal error: concurrent map iteration and map write
duncanista commented 2 days ago

Hey @kostaspap,

Thanks for reporting, we will assign someone shortly to look into this

In the meantime, could you try using v66 and let me know if the issue persists?

kostaspap commented 1 day ago

Hey @kostaspap,

Thanks for reporting, we will assign someone shortly to look into this

In the meantime, could you try using v66 and let me know if the issue persists?

Thanks for the quick response.

I didn't see anything related to the issue in the release notes for v66, but I will try it and report back if I see an improvement.

It's worth nothing that I haven't been able to reproduce easily, it seems to happen at random times for us.

duncanista commented 1 day ago

So, in the v66 release notes we only publish the relevant changes for Serverless in the Go side, some of those other bug fixes you might find them in the attached Datadog Agent tag – I know it's quite confusing, sorry about that! Since this problem seems to be affecting both Serverless, and their side, that's why we probably didn't announce it here.

I've already contacted the teams who own the change on the Go Agent side so we can see what might be happening there!

Thanks for the reply.