search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.72k stars 2.15k forks source link

[receiver/sqlquery] Integration test failing due to timeout #32657

Open crobert-1 opened 2 months ago

crobert-1 commented 2 months ago

Component(s)

receiver/sqlquery

Describe the issue you're reporting

Failing CI/CD link

Failure output:

Running target 'mod-integration-test' in module 'receiver/sqlqueryreceiver' as part of group 'receiver-3'
make --no-print-directory -C receiver/sqlqueryreceiver mod-integration-test
go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1
go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1
running go integration test ./... in /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/sqlqueryreceiver
/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/.tools/gotestsum --rerun-fails=1 --packages="./..." -- -race -timeout 360s -parallel 4 -tags=integration,"" -run=Integration
∅  internal/metadata
✖  . (6m0.087s)
ERROR rerun aborted because previous run had a suspected panic and some test may not have run

=== Skipped
=== SKIP: . TestOracleDBIntegrationMetrics (0.00s)
    integration_test.go:448: Skipping the test until https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/27577 is fixed

=== Failed
=== FAIL: .  (0.00s)
panic: test timed out after 6m0s
running tests:
    TestMysqlIntegrationMetrics (1m33s)

goroutine 551 [running]:
testing.(*M).startAlarm.func1()
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:2259 +0x259
created by time.goFunc
    /opt/hostedtoolcache/go/1.21.9/x64/src/time/sleep.go:176 +0x45

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00015f6c0, {0x3475af0, 0x1b}, 0x351e408)
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1649 +0x871
testing.runTests.func1(0x0?)
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:2054 +0x85
testing.tRunner(0xc00015f6c0, 0xc00089f938)
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1595 +0x262
testing.runTests(0xc0009ef5e0?, {0x58484e0, 0xf, 0xf}, {0xc0006ba3b8?, 0xc00089f9b8?, 0x585b740?})
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:2052 +0x8ae
testing.(*M).Run(0xc0009ef5e0)
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1925 +0xcd8
go.uber.org/goleak.VerifyTestMain({0x396c1e0, 0xc0009ef5e0}, {0xc000763e38, 0x1, 0x1})
    /home/runner/go/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x65
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlqueryreceiver.TestMain(0xb8235ea89a5b5fcf?)
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/sqlqueryreceiver/generated_package_test.go:12 +0x12b
main.main()
    _testmain.go:77 +0x308

goroutine 23 [IO wait, 6 minutes]:
internal/poll.runtime_pollWait(0x7f5632d38e28, 0x72)
    /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000477d20, 0xc000678480?, 0x0)
    /opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
    /opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).ReadMsg(0xc000477d00, {0xc000678480, 0x10, 0x10}, {0xc00048a020, 0x1000, 0x1000}, 0x4b0449?)
    /opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_unix.go:301 +0x55f
net.(*netFD).readMsg(0xc000477d00, {0xc000678480, 0x10, 0x10}, {0xc00048a020, 0x1000, 0x1000}, 0x1?)
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/fd_posix.go:78 +0xa9
net.(*UnixConn).readMsg(0xc00006e5f8, {0xc000678480, 0x10, 0x10}, {0xc00048a020, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/unixsock_posix.go:115 +0xb4
net.(*UnixConn).ReadMsgUnix(0xc00006e5f8, {0xc000678480, 0x10, 0x10}, {0xc00048a020, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/unixsock.go:143 +0xbb
github.com/godbus/dbus.(*oobReader).Read(0xc00048a000, {0xc000678480, 0x10, 0x10})
    /home/runner/go/pkg/mod/github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2/transport_unix.go:21 +0x99
io.ReadAtLeast({0x3972ae0, 0xc00048a000}, {0xc000678480, 0x10, 0x10}, 0x10)
    /opt/hostedtoolcache/go/1.21.9/x64/src/io/io.go:335 +0xd0
io.ReadFull(...)
    /opt/hostedtoolcache/go/1.21.9/x64/src/io/io.go:354
github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc000647380)
    /home/runner/go/pkg/mod/github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2/transport_unix.go:91 +0x288
github.com/godbus/dbus.(*Conn).inWorker(0xc00012aab0)
    /home/runner/go/pkg/mod/github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2/conn.go:294 +0x5f
created by github.com/godbus/dbus.(*Conn).Auth in goroutine 1
    /home/runner/go/pkg/mod/github.com/godbus/dbus@v0.0.0-20190726142602-4481cbc300e2/auth.go:118 +0xe74

goroutine 526 [select]:
github.com/stretchr/testify/assert.Eventually({0x396c180, 0xc0007b9520}, 0xc000d91720, 0xc0007db700?, 0x3138e40?, {0xc000d91740, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1897 +0x2b6
github.com/stretchr/testify/assert.Eventuallyf({0x396c180, 0xc0007b9520}, 0xc000d91720?, 0x0?, 0xc000d91720?, {0x3478666, 0x1c}, {0xc0007db700, 0x1, 0x1})
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertion_format.go:169 +0x198
github.com/stretchr/testify/require.Eventuallyf({0x3973ae0, 0xc0007b9520}, 0xc000463750?, 0x5fbb4a?, 0x4ae979?, {0x3478666, 0x1c}, {0xc0007db700, 0x1, 0x1})
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/require/require.go:468 +0xd8
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).createContainers.func1({{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:145 +0x2d1
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).createContainers in goroutine 525
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:143 +0x173

goroutine 546 [select]:
net/http.(*persistConn).writeLoop(0xc0004c30e0)
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transport.go:2421 +0x1bc
created by net/http.(*Transport).dialConn in goroutine 528
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transport.go:1777 +0x266b

goroutine 529 [select]:
net/http.(*persistConn).readLoop(0xc0004c30e0)
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transport.go:2238 +0x153d
created by net/http.(*Transport).dialConn in goroutine 528
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transport.go:1776 +0x25da

goroutine 525 [semacquire]:
sync.runtime_Semacquire(0xc000baf4e8?)
    /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc000baf4e0)
    /opt/hostedtoolcache/go/1.21.9/x64/src/sync/waitgroup.go:116 +0xa5
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).createContainers(0xc000b55900, 0xc0007b9520)
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:162 +0x40f
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).Run(0xc000b55900, 0xc0007b9520)
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:67 +0x65
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlqueryreceiver.TestMysqlIntegrationMetrics(0xc0007b9520?)
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/sqlqueryreceiver/integration_test.go:606 +0x865
testing.tRunner(0xc0007b9520, 0x351e408)
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 1
    /opt/hostedtoolcache/go/1.21.9/x64/src/testing/testing.go:1648 +0x846

goroutine 527 [IO wait]:
internal/poll.runtime_pollWait(0x7f5632d38c38, 0x72)
    /opt/hostedtoolcache/go/1.21.9/x64/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000b559a0, 0xc0005b5000?, 0x0)
    /opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
    /opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b55980, {0xc0005b5000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.21.9/x64/src/internal/poll/fd_unix.go:164 +0x405
net.(*netFD).Read(0xc000b55980, {0xc0005b5000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/fd_posix.go:55 +0x4b
net.(*conn).Read(0xc000a5c680, {0xc0005b5000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/net.go:179 +0xad
net/http.(*persistConn).Read(0xc0004c30e0, {0xc0005b5000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transport.go:1954 +0x105
bufio.(*Reader).fill(0xc000c1d860)
    /opt/hostedtoolcache/go/1.21.9/x64/src/bufio/bufio.go:113 +0x29a
bufio.(*Reader).ReadSlice(0xc000c1d860, 0x11?)
    /opt/hostedtoolcache/go/1.21.9/x64/src/bufio/bufio.go:379 +0x85
net/http/internal.readChunkLine(0xc0005b5000?)
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/internal/chunked.go:156 +0x34
net/http/internal.(*chunkedReader).beginChunk(0xc0007bd530)
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/internal/chunked.go:49 +0x5a
net/http/internal.(*chunkedReader).Read(0xc0007bd530, {0xc0008997c5, 0x183b, 0x183b})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/internal/chunked.go:125 +0x2b4
net/http.(*body).readLocked(0xc000b64a00, {0xc0008997c5, 0x183b, 0x183b})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transfer.go:839 +0xa9
net/http.(*body).Read(0xc000b64a00, {0xc0008997c5, 0x183b, 0x183b})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transfer.go:831 +0x185
net/http.(*bodyEOFSignal).Read(0xc000b64a40, {0xc0008997c5, 0x183b, 0x183b})
    /opt/hostedtoolcache/go/1.21.9/x64/src/net/http/transport.go:2810 +0x143
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*wrappedBody).Read(0xc000b64ac0, {0xc0008997c5, 0x183b, 0x183b})
    /home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/transport.go:250 +0x7a
io.ReadAll({0x7f55e8077368, 0xc0007f44c0})
    /opt/hostedtoolcache/go/1.21.9/x64/src/io/io.go:[704](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/8806392107/job/24171245484#step:5:705) +0x87
github.com/testcontainers/testcontainers-go.(*DockerProvider).attemptToPullImage(0xc00085cea0, {0x3988dd8?, 0x58913c0}, {0x3457127, 0xc}, {0x0, {0xc0000ae9c0, 0xd0}, 0x0, {0x0, ...}})
    /home/runner/go/pkg/mod/github.com/testcontainers/testcontainers-go@v0.30.0/docker.go:1283 +0x8bb
github.com/testcontainers/testcontainers-go.(*DockerProvider).CreateContainer(_, {_, _}, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...})
    /home/runner/go/pkg/mod/github.com/testcontainers/testcontainers-go@v0.30.0/docker.go:1047 +0x1a54
github.com/testcontainers/testcontainers-go.GenericContainer({_, _}, {{{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...})
    /home/runner/go/pkg/mod/github.com/testcontainers/testcontainers-go@v0.30.0/generic.go:73 +0x350
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).createContainers.func1.1()
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:146 +0xcf
github.com/stretchr/testify/assert.Eventually.func1()
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1902 +0x34
created by github.com/stretchr/testify/assert.Eventually in goroutine 526
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1902 +0x3b6

goroutine 550 [chan receive]:
github.com/testcontainers/testcontainers-go.(*Reaper).Connect.func1({0x3995ab8?, 0xc000a5c6b0})
    /home/runner/go/pkg/mod/github.com/testcontainers/testcontainers-go@v0.30.0/reaper.go:358 +0xa0e
created by github.com/testcontainers/testcontainers-go.(*Reaper).Connect in goroutine 527
    /home/runner/go/pkg/mod/github.com/testcontainers/testcontainers-go@v0.30.0/reaper.go:323 +0x2ec
FAIL    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlqueryreceiver 360.087s

=== FAIL: . TestMysqlIntegrationMetrics (unknown)

DONE 5 tests, 1 skipped, 2 failures in 399.971s
make[2]: *** [../../Makefile.Common:142: mod-integration-test] Error 3
make[1]: *** [Makefile:169: receiver/sqlqueryreceiver] Error 2
make: *** [Makefile:126: gointegration-test] Error 2
make[1]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib'
github-actions[bot] commented 2 months ago

Pinging code owners for receiver/sqlquery: @dmitryax @crobert-1. See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 commented 2 months ago

It looks like the main culprit here may be the timeout hit when attempting to create the container for the test.

goroutine 526 [select]:
github.com/stretchr/testify/assert.Eventually({0x396c180, 0xc0007b9520}, 0xc000d91720, 0xc0007db700?, 0x3138e40?, {0xc000d91740, 0x2, 0x2})
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertions.go:1897 +0x2b6
github.com/stretchr/testify/assert.Eventuallyf({0x396c180, 0xc0007b9520}, 0xc000d91720?, 0x0?, 0xc000d91720?, {0x3478666, 0x1c}, {0xc0007db700, 0x1, 0x1})
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/assert/assertion_format.go:169 +0x198
github.com/stretchr/testify/require.Eventuallyf({0x3973ae0, 0xc0007b9520}, 0xc000463750?, 0x5fbb4a?, 0x4ae979?, {0x3478666, 0x1c}, {0xc0007db700, 0x1, 0x1})
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.9.0/require/require.go:468 +0xd8
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).createContainers.func1({{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:145 +0x2d1
created by github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest.(*IntegrationTest).createContainers in goroutine 525
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/coreinternal/scraperinttest/scraperint.go:143 +0x173

The default timeout for creating a container in the method is 5 minutes. The integration tests themselves in contrib have a 6 minute timeout, so we could potentially increase this default value.

atoulme commented 2 months ago

This might be tied to throttling of Docker image pulls.

rogercoll commented 1 month ago

Related issues: