search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.92k stars 2.28k forks source link

Flaky test - dockerstatsreceiver TestDefaultMetricsIntegration #5983

Closed jpkrohling closed 1 year ago

jpkrohling commented 2 years ago

Seen here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/4038604844?check_suite_focus=true

=== RUN   TestDefaultMetricsIntegration
    container.go:112: image pull: {"status":"Pulling from library/nginx","id":"1.17"}
    container.go:112: image pull: {"status":"Pulling fs layer","progressDetail":{},"id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Pulling fs layer","progressDetail":{},"id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Pulling fs layer","progressDetail":{},"id":"11fa52a0fdc0"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":253135,"total":23922586},"progress":"[\u003e                                                  ]  253.1kB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":751980,"total":23922586},"progress":"[=\u003e                                                 ]    752kB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":2238828,"total":23922586},"progress":"[====\u003e                                              ]  2.239MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":4954476,"total":23922586},"progress":"[==========\u003e                                        ]  4.954MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":5945708,"total":23922586},"progress":"[============\u003e                                      ]  5.946MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":7674220,"total":23922586},"progress":"[================\u003e                                  ]  7.674MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":10373484,"total":23922586},"progress":"[=====================\u003e                             ]  10.37MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":203,"total":203},"progress":"[==================================================\u003e]     203B/203B","id":"11fa52a0fdc0"}
    container.go:112: image pull: {"status":"Verifying Checksum","progressDetail":{},"id":"11fa52a0fdc0"}
    container.go:112: image pull: {"status":"Download complete","progressDetail":{},"id":"11fa52a0fdc0"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":277701,"total":27098756},"progress":"[\u003e                                                  ]  277.7kB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":11356524,"total":23922586},"progress":"[=======================\u003e                           ]  11.36MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":13330796,"total":23922586},"progress":"[===========================\u003e                       ]  13.33MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":559467,"total":27098756},"progress":"[=\u003e                                                 ]  559.5kB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":15800684,"total":23922586},"progress":"[=================================\u003e                 ]   15.8MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":18020716,"total":23922586},"progress":"[=====================================\u003e             ]  18.02MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":19741036,"total":23922586},"progress":"[=========================================\u003e         ]  19.74MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":21465452,"total":23922586},"progress":"[============================================\u003e      ]  21.47MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":837995,"total":27098756},"progress":"[=\u003e                                                 ]    838kB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":22694252,"total":23922586},"progress":"[===============================================\u003e   ]  22.69MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":1116523,"total":27098756},"progress":"[==\u003e                                                ]  1.117MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":1395051,"total":27098756},"progress":"[==\u003e                                                ]  1.395MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":23922586,"total":23922586},"progress":"[==================================================\u003e]  23.92MB/23.92MB","id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Download complete","progressDetail":{},"id":"b90c53a0b692"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":1673579,"total":27098756},"progress":"[===\u003e                                               ]  1.674MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":1952107,"total":27098756},"progress":"[===\u003e                                               ]  1.952MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":2230635,"total":27098756},"progress":"[====\u003e                                              ]  2.231MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":2509163,"total":27098756},"progress":"[====\u003e                                              ]  2.509MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":2787691,"total":27098756},"progress":"[=====\u003e                                             ]  2.788MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":3066219,"total":27098756},"progress":"[=====\u003e                                             ]  3.066MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":3344747,"total":27098756},"progress":"[======\u003e                                            ]  3.345MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":3623275,"total":27098756},"progress":"[======\u003e                                            ]  3.623MB/27.1MB","id":"afb6ec6fdc1c"}
    container.go:112: image pull: {"status":"Downloading","progressDetail":{"current":3901803,"total":27098756},"progress":"[=======\u003e                                           ]  3.902MB/27.1MB","id":"afb6ec6fdc1c"}
    /opt/hostedtoolcache/go/1.17.2/x64/src/internal/poll/fd_unix.go:167 +0x419
net.(*netFD).Read(0xc00047e180, {0xc00047d000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/fd_posix.go:56 +0x51
net.(*conn).Read(0xc00009e4b0, {0xc00047d000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/net.go:183 +0xb1
net/http.(*persistConn).Read(0xc0000f7680, {0xc00047d000, 0x1000, 0x1000})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transport.go:1926 +0x110
bufio.(*Reader).fill(0xc000383440)
    /opt/hostedtoolcache/go/1.17.2/x64/src/bufio/bufio.go:101 +0x294
bufio.(*Reader).ReadSlice(0xc000383440, 0xd8)
    /opt/hostedtoolcache/go/1.17.2/x64/src/bufio/bufio.go:360 +0x89
net/http/internal.readChunkLine(0x1)
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/internal/chunked.go:122 +0x36
net/http/internal.(*chunkedReader).beginChunk(0xc00007c1e0)
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/internal/chunked.go:48 +0x45
net/http/internal.(*chunkedReader).Read(0xc00007c1e0, {0xc0004ae52b, 0xad5, 0xad5})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/internal/chunked.go:93 +0x2ad
net/http.(*body).readLocked(0xc000072080, {0xc0004ae52b, 0xad5, 0xad5})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transfer.go:843 +0xb8
net/http.(*body).Read(0xc000072080, {0xc0004ae52b, 0xad5, 0xad5})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transfer.go:835 +0x17e
net/http.(*bodyEOFSignal).Read(0xc0000720c0, {0xc0004ae52b, 0xad5, 0xad5})
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transport.go:2768 +0x143
bufio.(*Scanner).Scan(0xc0000a3a30)
    /opt/hostedtoolcache/go/1.17.2/x64/src/bufio/scan.go:215 +0x13d6
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/testing/container.(*Containers).pullImage(0xc00039f040, {0xe8c4a9, 0x1c})
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/common/testing/container/container.go:111 +0x457
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/testing/container.(*Containers).StartImageWithEnv(0xc00039f040, {0xe8c4a9, 0x1c}, {0x0, 0x0, 0x0}, {0xc0000a3dc0, 0x1, 0x17})
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/common/testing/container/container.go:191 +0xc5
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common/testing/container.(*Containers).StartImage(...)
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/internal/common/testing/container/container.go:187
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver.TestDefaultMetricsIntegration(0xc00046c4e0)
    /home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/dockerstatsreceiver/integration_test.go:71 +0x1bc
testing.tRunner(0xc00046c4e0, 0xeaac00)
    /opt/hostedtoolcache/go/1.17.2/x64/src/testing/testing.go:1259 +0x230
created by testing.(*T).Run
    /opt/hostedtoolcache/go/1.17.2/x64/src/testing/testing.go:1306 +0x727

goroutine 22 [select]:
net/http.(*persistConn).readLoop(0xc0000f7680)
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transport.go:2207 +0x14ef
created by net/http.(*Transport).dialConn
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transport.go:1747 +0x2ef8

goroutine 23 [select]:
net/http.(*persistConn).writeLoop(0xc0000f7680)
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transport.go:2386 +0x1a9
created by net/http.(*Transport).dialConn
    /opt/hostedtoolcache/go/1.17.2/x64/src/net/http/transport.go:1748 +0x2f8b
FAIL    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/dockerstatsreceiver  60.039s
FAIL
make[2]: *** [../../Makefile.Common:56: do-integration-tests-with-cover] Error 1
make[2]: Leaving directory '/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/dockerstatsreceiver'
make[1]: *** [Makefile:148: for-all-target-receiver/dockerstatsreceiver] Error 2
make: *** [Makefile:48: integration-tests-with-cover] Error 2
rmfitzpatrick commented 2 years ago

Looks like long pull times from docker hub. @jpkrohling do you happen to know if the docker login secrets are available to the build-and-test integration-tests* job: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/ca4bfe1c4a9305a291397fc0ae2810ef5c7da039/.github/workflows/build-and-test.yml#L406? I think adding caching is one way to help with this. We could also run a local registry and use the actions/cache action: https://github.com/dtinth/github-actions-docker-layer-caching-poc/blob/210fdfb3f3b536a3e5dbea37da5f1bb8b285f84b/.github/workflows/dockerimage.yml#L33

jpkrohling commented 2 years ago

do you happen to know if the docker login secrets are available to the build-and-test integration-tests* job:

There are no such secrets defined for this project. Should I just set my own credentials for now?

codeboten commented 2 years ago

Decided to test out using ghcr.io instead to see if it makes a difference. see https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/6110. Not sure if this will make a difference or not, since pulling from dockerhub usually happens within seconds, maybe something else is going on there..

jpkrohling commented 2 years ago

I'll set my own credentials for now, and let's see how it compares. If it still causes a problem, we should definitely go for your proposal.

jpkrohling commented 2 years ago

Both DOCKER_USERNAME and DOCKER_PASSWORD are now available as a secret.

djaglowski commented 2 years ago

Here again: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/6043053360?check_suite_focus=true#step:7:161

github-actions[bot] commented 1 year ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

github-actions[bot] commented 1 year ago

Pinging code owners: @rmfitzpatrick @jamesmoessis. See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 1 year ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 1 year ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

atoulme commented 1 year ago

@codeboten do you still want to change the docker registry? I notice that the PR was closed as inactive. I recommend we close this bug as we don't have any reports for the last year.

github-actions[bot] commented 1 year ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 1 year ago

This issue has been closed as inactive because it has been stale for 120 days with no activity.