search

testcontainers / testcontainers-go

Testcontainers for Go is a Go package that makes it simple to create and clean up container-based dependencies for automated integration/smoke tests. The clean, easy-to-use API enables developers to programmatically define containers that should be run as part of a test and clean up those resources when the test is done.
https://golang.testcontainers.org
MIT License
3.68k stars 504 forks source link

[Bug]: random clickhouse test failures #2848

Open stevenh opened 1 month ago

stevenh commented 1 month ago

Testcontainers version

0.34.0

Using the latest Testcontainers version?

Yes

Host OS

N/A

Host arch

N/A

Go version

N/A

Docker version

N/A

Docker info

N/A

What happened?

Tests are randomly failing in CI with clickhouse_test.go:290: [handshake] unexpected packet [72] from server see: https://github.com/testcontainers/testcontainers-go/actions/runs/11478021546/job/31941936972?pr=2846

Relevant log output

=== FAIL: . TestClickHouseWithZookeeper (unknown) 2024/10/23 11:11:19 🐳 Creating container for image zookeeper:3.7 2024/10/23 11:11:19 ✅ Container created: 3205da1f43da 2024/10/23 11:11:19 🐳 Starting container: 3205da1f43da 2024/10/23 11:11:19 ✅ Container started: 3205da1f43da 2024/10/23 11:11:19 ⏳ Waiting for container id 3205da1f43da image: zookeeper:3.7. Waiting for: &{Port:2181/tcp timeout: PollInterval:100ms skipInternalCheck:false} 2024/10/23 11:11:19 🔔 Container is ready: 3205da1f43da 2024/10/23 11:11:19 🐳 Creating container for image clickhouse/clickhouse-server:23.3.8.21-alpine 2024/10/23 11:11:20 ✅ Container created: 2916bb606516 2024/10/23 11:11:20 🐳 Starting container: 2916bb606516 2024/10/23 11:11:20 ✅ Container started: 2916bb606516 2024/10/23 11:11:20 ⏳ Waiting for container id 2916bb606516 image: clickhouse/clickhouse-server:23.3.8.21-alpine. Waiting for: &{timeout: deadline: Strategies:[0xc00052ba70]} 2024/10/23 11:11:23 🔔 Container is ready: 2916bb606516 clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server coverage: 85.0% of statements ERROR rerun aborted because previous run had a suspected panic and some test may not have run make: *** [../../commons-test.mk:42: test-unit] Error 3

Additional information

No response

laskoviymishka commented 4 weeks ago

Logs in clickhouse:


  2024.10.28 15:34:42.570353 [ 66 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
  2024.10.28 15:34:42.733650 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:8123 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.733819 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9000 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.733984 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9004 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.734141 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9005 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9005 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.734348 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

was able to repro it in CI

stevenh commented 2 weeks ago

Another incident of this here

FAIL TestClickHouseWithInitScripts (5.36s)
  2024/11/05 19:04:55 🐳 Creating container for image clickhouse/clickhouse-server:23.3.8.21-alpine
  2024/11/05 19:04:55 ✅ Container created: 88b56ce7d3a5
  2024/11/05 19:04:55 🐳 Starting container: 88b56ce7d3a5
  2024/11/05 19:04:55 ✅ Container started: 88b56ce7d3a5
  2024/11/05 19:04:55 ⏳ Waiting for container id 88b56ce7d3a5 image: clickhouse/clickhouse-server:23.3.8.21-alpine. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0xc0003aa7e0]}
  2024/11/05 19:04:58 🔔 Container is ready: 88b56ce7d3a5
      clickhouse_test.go:155: 
            Error Trace:    /home/runner/work/testcontainers-go/testcontainers-go/modules/clickhouse/clickhouse_test.go:155
            Error:          Received unexpected error:
                            [handshake] unexpected packet [72] from server
            Test:           TestClickHouseWithInitScripts
  2024/11/05 19:04:58 🐳 Stopping container: 88b56ce7d3a5
  2024/11/05 19:05:00 ✅ Container stopped: 88b56ce7d3a5
  2024/11/05 19:05:00 🐳 Terminating container: 88b56ce7d3a5
  2024/11/05 19:05:00 🚫 Container terminated: 88b56ce7d3a5