search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.9k stars 2.27k forks source link

[internal/otelarrow] Flaky TestMRUReset #34252

Open djaglowski opened 1 month ago

djaglowski commented 1 month ago

Component(s)

internal/otelarrow

What happened?

Failing here and on main.

Likely introduced in https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/33579

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

=== FAIL: compression/zstd TestMRUReset (0.00s)
    mru_test.go:83: 
            Error Trace:    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/compression/zstd/mru_test.go:83
            Error:          Not equal: 
                            expected: 0
                            actual  : 1
            Test:           TestMRUReset

=== FAIL: test  (0.00s)
panic: test timed out after 10m0s
running tests:
    TestIntegrationSelfTracing (9m30s)

goroutine 66621 [running]:
testing.(*M).startAlarm.func1()
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:2259 +0x259
created by time.goFunc
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/time/sleep.go:176 +0x45

goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0003041a0, {0x1416bb7dd, 0x1a}, 0x141713ec8)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1649 +0x871
testing.runTests.func1(0x0?)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:2054 +0x85
testing.tRunner(0xc0003041a0, 0xc00051faf8)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1595 +0x262
testing.runTests(0xc000360f00?, {0x1413ffd40, 0x3, 0x3}, {0x1c?, 0x1e?, 0x1423243e0?})
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:2052 +0x8ae
testing.(*M).Run(0xc000360f00)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1925 +0xcd8
main.main()
    _testmain.go:51 +0x2be

goroutine 28393 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc042bf0f48?)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc042bf0f40)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.testIntegrationTraces({0x14199c020?, 0xc052cb0f50}, 0xc000380ea0, {0x141a6d1cf?, 0xf?}, 0x141c727bb?, 0x141714008, 0x141714020, 0x141714038)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:218 +0x972
github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.TestIntegrationSelfTracing(0x0?)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:476 +0xbf
testing.tRunner(0xc000380ea0, 0x141713ec8)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 1
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/testing/testing.go:1648 +0x846

goroutine 28398 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc011e0c1d8?)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc011e0c1d0)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
google.golang.org/grpc.(*Server).stop(0xc011e0c000, 0x1)
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1921 +0x4ac
google.golang.org/grpc.(*Server).GracefulStop(...)
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1883
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver.(*otelArrowReceiver).Shutdown(0xc0000e33b0, {0xc0009e5d68?, 0x1400511b4?})
ERROR rerun aborted because previous run had a suspected panic and some test may not have run
make[2]: *** [../../Makefile.Common:126: test] Error 3
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/otelarrow.go:172 +0x5f
github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent.(*SharedComponent).Shutdown.func1()
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/sharedcomponent/sharedcomponent.go:72 +0x6d
sync.(*Once).doSlow(0xc000d87b7c, 0xc0009e5ec0)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/once.go:74 +0xf1
sync.(*Once).Do(0xc000d87b7c, 0xc0009e5eb8?)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/once.go:65 +0x45
github.com/open-telemetry/opentelemetry-collector-contrib/internal/sharedcomponent.(*SharedComponent).Shutdown(0xc000d87b60, {0x14199c020?, 0xc052cb0f50})
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/sharedcomponent/sharedcomponent.go:71 +0xb9
github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.testIntegrationTraces.func1()
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:179 +0x172
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test.testIntegrationTraces in goroutine 28393
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/otelarrow/test/e2e_test.go:175 +0x3ab

goroutine 66582 [chan send, 9 minutes]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).replyToCaller(...)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:485
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).consumeDone(0xc001598180, {0x14199bfe8, 0xc001111380}, 0xc008b39f10)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:480 +0x2e7
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).consumeAndRespond(0xc001f908c0, {0x14199bfe8?, 0xc001111380}, {0x14148ab40, 0xc02d5345a0}, 0xc001598180)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:659 +0x1fe
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).recvOne in goroutine 60128
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:642 +0xe5d

goroutine 60127 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc03dbb94c8?)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc03dbb94c0)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).anyStream(0xc001f908c0, {0x47fd8bc0, 0xc0550655a0}, {0xc0001be140, 0x49})
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:405 +0xeac
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).ArrowTraces(0x140d8bd25?, {0x1419a2978?, 0xc0550655a0})
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:309 +0x78
github.com/open-telemetry/otel-arrow/api/experimental/arrow/v1._ArrowTracesService_ArrowTraces_Handler({0x141619820?, 0xc001f908c0}, {0x14199f9f0?, 0xc040565220})
    C:/Users/runneradmin/go/pkg/mod/github.com/open-telemetry/otel-arrow@v0.24.0/api/experimental/arrow/v1/arrow_service_grpc.pb.go:122 +0xb6
go.opentelemetry.io/collector/config/configgrpc.(*ServerConfig).toServerOption.enhanceStreamWithClientInformation.func10({0x141619820, 0xc001f908c0}, {0x1419a0098, 0xc0000e21e0}, 0x14147bf40?, 0x141715ce8)
    C:/Users/runneradmin/go/pkg/mod/go.opentelemetry.io/collector/config/configgrpc@v0.105.1-0.20240717163034-43ed6184f9fe/configgrpc.go:451 +0x87
google.golang.org/grpc.(*Server).processStreamingRPC(0xc011e0c000, {0x14199bfe8, 0xc0008db5c0}, {0x1419a3500, 0xc000c6a300}, 0xc0007dd7a0, 0xc040e3e360, 0x1413ed640, 0x0)
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1680 +0x1f51
google.golang.org/grpc.(*Server).handleStream(0xc011e0c000, {0x1419a3500, 0xc000c6a300}, 0xc0007dd7a0)
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1794 +0x1a25
google.golang.org/grpc.(*Server).serveStreams.func2.1()
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029 +0x15d
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 57553
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1040 +0x225

goroutine 60129 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc0000170c0?)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc0000170b8)
    C:/hostedtoolcache/windows/go/1.21.12/x64/src/sync/waitgroup.go:116 +0xa8
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).flushSender(0xc0000170b0, {0x47fd8bc0, 0xc0550655a0}, 0x0?, 0x0?)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:726 +0x58
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).srvSendLoop(0x0?, {0x14199c020, 0xc000aa90e0}, {0x47fd8bc0, 0xc0550655a0}, 0x0?, 0xc05400d560)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:745 +0x177
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).anyStream.func3()
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:394 +0x1ce
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).anyStream in goroutine 60127
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:388 +0x985

goroutine 28366 [chan receive, 9 minutes]:
google.golang.org/grpc.(*Server).Serve.func1()
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:855 +0xad
google.golang.org/grpc.(*Server).Serve(0xc011e0c000, {0x141999210?, 0xc040e34780})
    C:/Users/runneradmin/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:915 +0xd73
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver.(*otelArrowReceiver).startGRPCServer.func1()
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/otelarrow.go:91 +0xf1
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver.(*otelArrowReceiver).startGRPCServer in goroutine 28398
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/otelarrow.go:88 +0x2a9

goroutine 66584 [chan send, 9 minutes]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).replyToCaller(...)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:485
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*inFlightData).consumeDone(0xc001599500, {0x14199bfe8, 0xc001111950}, 0xc002ed5f10)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:480 +0x2e7
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*Receiver).consumeAndRespond(0xc001f908c0, {0x14199bfe8?, 0xc001111950}, {0x14148ab40, 0xc02d535368}, 0xc001599500)
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:659 +0x1fe
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow.(*receiverStream).recvOne in goroutine 60128
    D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/otelarrowreceiver/internal/arrow/arrow.go:642 +0xe5d
FAIL    github.com/open-telemetry/opentelemetry-collector-contrib/internal/otelarrow/test   600.474s


### Additional context

_No response_
github-actions[bot] commented 1 month ago

Pinging code owners:

pjanotti commented 1 month ago

[EDIT: removed the comment here and moved it to issue #34719]

pjanotti commented 1 month ago

[EDIT: removed the comment here and moved it to issue https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/34719]

axw commented 1 month ago

@pjanotti are you referring to any/all flaky tests here? I don't see any mention of TestMRUReset in those failures. I think we should close this issue and create new ones for other flaky tests.

pjanotti commented 4 weeks ago

@axw yeah, makes more sense, moved to #34719