search

apache / shardingsphere

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Apache License 2.0
20.01k stars 6.76k forks source link

Integration tests occasionally stuck in waiting for container ready #19648

Closed TeslaCN closed 2 years ago

TeslaCN commented 2 years ago

Bug Report

Which version of ShardingSphere did you use?

master - 9284c595a727ca70b845ad7ec4792cc549db7fa7

ShardingSphere-Proxy integration tests

Maybe this is similar to

Environment

~ ➤ docker info                                                                                                                                                                                                                                                                          
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 16
  Running: 4
  Paused: 0
  Stopped: 12
 Images: 160
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.0-41-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.25GiB
 Name: wuweijie-ubuntu
 ID: NTWV:UXYU:53NR:XS22:OJ7L:BVPD:V4KW:ZUEH:6RJQ:7BNV:7KGN:VZ5Y
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: shardingsphere
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://gbkpkqgm.mirror.aliyuncs.com/
 Live Restore Enabled: false

Expected behavior

Tests could be finished.

Actual behavior

Container ZooKeeper has readied for tens of minutes.

image

image

Logs of maven:

[INFO ] 2022-07-28 16:58:02.510 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,510 [myid:1] - INFO  [main:ZKDatabase@152] - zookeeper.commitLogCount=500
[INFO ] 2022-07-28 16:58:02.514 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,514 [myid:1] - INFO  [main:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED
[INFO ] 2022-07-28 16:58:02.514 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,514 [myid:1] - INFO  [main:FileTxnSnapLog@470] - Snapshotting: 0x0 to /data/version-2/snapshot.0
[INFO ] 2022-07-28 16:58:02.516 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,516 [myid:1] - INFO  [main:ZKDatabase@289] - Snapshot loaded in 6 ms, highest zxid is 0x0, digest is 1371985504
[INFO ] 2022-07-28 16:58:02.516 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,516 [myid:1] - INFO  [main:FileTxnSnapLog@470] - Snapshotting: 0x0 to /data/version-2/snapshot.0
[INFO ] 2022-07-28 16:58:02.517 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,516 [myid:1] - INFO  [main:ZooKeeperServer@529] - Snapshot taken in 0 ms
[INFO ] 2022-07-28 16:58:02.522 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,522 [myid:1] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@136] - PrepRequestProcessor (sid:0) started, reconfigEnabled=false
[INFO ] 2022-07-28 16:58:02.522 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,522 [myid:1] - INFO  [main:RequestThrottler@74] - zookeeper.request_throttler.shutdownTimeout = 10000
[INFO ] 2022-07-28 16:58:02.531 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,530 [myid:1] - INFO  [main:ContainerManager@83] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0
[INFO ] 2022-07-28 16:58:02.531 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 08:58:02,531 [myid:1] - INFO  [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.
[INFO ] 2022-07-28 17:18:54.509 [docker-java-stream-2041008728] dbtbl_with_readwrite_splitting:zookeeper - STDOUT: 2022-07-28 09:18:54,508 [myid:1] - INFO  [SyncThread:0:FileTxnLog@284] - Creating new log file: log.1

But the surefire process was still waiting for ready:

{} ~ jstack 2290346
2022-07-28 17:17:22
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0.1+12-LTS-39 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fec54001f60, length=18, elements={
0x00007feca80259b0, 0x00007feca8164e40, 0x00007feca8166220, 0x00007feca816b580,
0x00007feca816c930, 0x00007feca816dd40, 0x00007feca816f770, 0x00007feca8170ca0,
0x00007feca8172110, 0x00007feca817d960, 0x00007feca8c02320, 0x00007feca8c77c40,
0x00007feca8cdfc00, 0x00007febec016eb0, 0x00007febdc089ad0, 0x00007feca92711d0,
0x00007feca927f250, 0x00007fec54001550
}

"main" #1 prio=5 os_prio=0 cpu=1161262.57ms elapsed=1164.15s tid=0x00007feca80259b0 nid=0x22f2ae runnable  [0x00007fecadfb5000]
   java.lang.Thread.State: RUNNABLE
    at org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
    at org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
    at org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
    at org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
    at org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
    at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
    at org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
    at org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
    at org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
    at org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$178/0x0000000800ec12b0.close(Unknown Source)
    at com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
    at org.testcontainers.containers.output.FrameConsumerResultCallback.close(FrameConsumerResultCallback.java:100)
    at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:51)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:892)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:440)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
    at org.testcontainers.containers.GenericContainer$$Lambda$194/0x0000000800ece220.call(Unknown Source)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
    at org.apache.shardingsphere.test.integration.env.container.atomic.DockerITContainer.start(DockerITContainer.java:49)
    at org.apache.shardingsphere.test.integration.env.container.atomic.ITContainers$$Lambda$95/0x0000000800d9ac08.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.base@17.0.1/ForEachOps.java:183)
    at java.util.stream.ReferencePipeline$2$1.accept(java.base@17.0.1/ReferencePipeline.java:179)
    at java.util.LinkedList$LLSpliterator.forEachRemaining(java.base@17.0.1/LinkedList.java:1242)
    at java.util.stream.AbstractPipeline.copyInto(java.base@17.0.1/AbstractPipeline.java:509)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@17.0.1/AbstractPipeline.java:499)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(java.base@17.0.1/ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.{} ~ jstack 2290346
2022-07-28 17:17:22
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0.1+12-LTS-39 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fec54001f60, length=18, elements={
0x00007feca80259b0, 0x00007feca8164e40, 0x00007feca8166220, 0x00007feca816b580,
0x00007feca816c930, 0x00007feca816dd40, 0x00007feca816f770, 0x00007feca8170ca0,
0x00007feca8172110, 0x00007feca817d960, 0x00007feca8c02320, 0x00007feca8c77c40,
0x00007feca8cdfc00, 0x00007febec016eb0, 0x00007febdc089ad0, 0x00007feca92711d0,
0x00007feca927f250, 0x00007fec54001550
}

"main" #1 prio=5 os_prio=0 cpu=1161262.57ms elapsed=1164.15s tid=0x00007feca80259b0 nid=0x22f2ae runnable  [0x00007fecadfb5000]
   java.lang.Thread.State: RUNNABLE
    at org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
    at org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:310)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
    at org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
    at org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
    at org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
    at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
    at org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
    at org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
    at org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
    at org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$178/0x0000000800ec12b0.close(Unknown Source)
    at com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
    at org.testcontainers.containers.output.FrameConsumerResultCallback.close(FrameConsumerResultCallback.java:100)
    at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:51)
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:892)
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:440)
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
    at org.testcontainers.containers.GenericContainer$$Lambda$194/0x0000000800ece220.call(Unknown Source)
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
    at org.apache.shardingsphere.test.integration.env.container.atomic.DockerITContainer.start(DockerITContainer.java:49)
    at org.apache.shardingsphere.test.integration.env.container.atomic.ITContainers$$Lambda$95/0x0000000800d9ac08.accept(Unknown Source)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.base@17.0.1/ForEachOps.java:183)
    at java.util.stream.ReferencePipeline$2$1.accept(java.base@17.0.1/ReferencePipeline.java:179)
    at java.util.LinkedList$LLSpliterator.forEachRemaining(java.base@17.0.1/LinkedList.java:1242)
    at java.util.stream.AbstractPipeline.copyInto(java.base@17.0.1/AbstractPipeline.java:509)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@17.0.1/AbstractPipeline.java:499)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(java.base@17.0.1/ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(java.base@17.0.1/ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(java.base@17.0.1/AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(java.base@17.0.1/ReferencePipeline.java:596)
    at org.apache.shardingsphere.test.integration.env.container.atomic.ITContainers.start(ITContainers.java:82)
    - locked <0x000000061cefdc20> (a org.apache.shardingsphere.test.integration.env.container.atomic.ITContainers)
    at org.apache.shardingsphere.test.integration.container.compose.mode.ClusterComposedContainer.start(ClusterComposedContainer.java:64)
    at org.apache.shardingsphere.test.integration.engine.BaseITCase.setUp(BaseITCase.java:78)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.1/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.1/NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.1/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.1/Method.java:568)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.73ms elapsed=1164.12s tid=0x00007feca8164e40 nid=0x22f2b5 waiting on condition  [0x00007fec71142000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ref.Reference.waitForReferencePendingList(java.base@17.0.1/Native Method)
    at java.lang.ref.Reference.processPendingReferences(java.base@17.0.1/Reference.java:253)
    at java.lang.ref.Reference$ReferenceHandler.run(java.base@17.0.1/Reference.java:215)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.53ms elapsed=1164.12s tid=0x00007feca8166220 nid=0x22f2b6 in Object.wait()  [0x00007fec707fe000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@17.0.1/ReferenceQueue.java:155)
    - locked <0x000000061c5b4578> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(java.base@17.0.1/ReferenceQueue.java:176)
    at java.lang.ref.Finalizer$FinalizerThread.run(java.base@17.0.1/Finalizer.java:172)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.22ms elapsed=1164.12s tid=0x00007feca816b580 nid=0x22f2b7 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.57ms elapsed=1164.12s tid=0x00007feca816c930 nid=0x22f2b8 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #6 daemon prio=9 os_prio=0 cpu=13.62ms elapsed=1164.12s tid=0x00007feca816dd40 nid=0x22f2b9 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=1485.55ms elapsed=1164.12s tid=0x00007feca816f770 nid=0x22f2ba waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #15 daemon prio=9 os_prio=0 cpu=489.43ms elapsed=1164.12s tid=0x00007feca8170ca0 nid=0x22f2bb waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #19 daemon prio=9 os_prio=0 cpu=16.61ms elapsed=1164.12s tid=0x00007feca8172110 nid=0x22f2bc runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #20 daemon prio=8 os_prio=0 cpu=1.42ms elapsed=1164.11s tid=0x00007feca817d960 nid=0x22f2bd in Object.wait()  [0x00007fec5b78b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@17.0.1/ReferenceQueue.java:155)
    - locked <0x000000061c5c00f8> (a java.lang.ref.ReferenceQueue$Lock)
    at jdk.internal.ref.CleanerImpl.run(java.base@17.0.1/CleanerImpl.java:140)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)
    at jdk.internal.misc.InnocuousThread.run(java.base@17.0.1/InnocuousThread.java:162)

"Notification Thread" #22 daemon prio=9 os_prio=0 cpu=0.06ms elapsed=1163.86s tid=0x00007feca8c02320 nid=0x22f2cb runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"surefire-forkedjvm-command-thread" #23 daemon prio=5 os_prio=0 cpu=14.96ms elapsed=1163.80s tid=0x00007feca8c77c40 nid=0x22f2cf runnable  [0x00007fec5a484000]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(java.base@17.0.1/Native Method)
    at java.io.FileInputStream.read(java.base@17.0.1/FileInputStream.java:276)
    at java.io.BufferedInputStream.fill(java.base@17.0.1/BufferedInputStream.java:244)
    at java.io.BufferedInputStream.read(java.base@17.0.1/BufferedInputStream.java:263)
    - locked <0x000000061c5b9378> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readInt(java.base@17.0.1/DataInputStream.java:393)
    at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
    at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"surefire-forkedjvm-ping-30s" #24 daemon prio=5 os_prio=0 cpu=746.88ms elapsed=1163.76s tid=0x00007feca8cdfc00 nid=0x22f2d0 waiting on condition  [0x00007fec5a383000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@17.0.1/Native Method)
    - parking to wait for  <0x000000061cc01470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.1/LockSupport.java:252)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@17.0.1/AbstractQueuedSynchronizer.java:1672)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.1/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.1/ThreadPoolExecutor.java:1062)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1122)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"process reaper" #25 daemon prio=10 os_prio=0 cpu=271.39ms elapsed=1163.74s tid=0x00007febec016eb0 nid=0x22f2d3 waiting on condition  [0x00007fec700f7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@17.0.1/Native Method)
    - parking to wait for  <0x000000061cc02490> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.1/LockSupport.java:252)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@17.0.1/SynchronousQueue.java:401)
    at java.util.concurrent.SynchronousQueue.poll(java.base@17.0.1/SynchronousQueue.java:903)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.1/ThreadPoolExecutor.java:1061)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1122)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"OkHttp ConnectionPool" #28 daemon prio=5 os_prio=0 cpu=34.74ms elapsed=1162.53s tid=0x00007febdc089ad0 nid=0x22f2db in Object.wait()  [0x00007fec58be2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.Object.wait(java.base@17.0.1/Object.java:472)
    at org.testcontainers.shaded.okhttp3.internal.connection.RealConnectionPool.lambda$new$0(RealConnectionPool.java:62)
    - locked <0x000000061cd9a6d0> (a org.testcontainers.shaded.okhttp3.internal.connection.RealConnectionPool)
    at org.testcontainers.shaded.okhttp3.internal.connection.RealConnectionPool$$Lambda$129/0x0000000800db4438.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"testcontainers-ryuk" #31 daemon prio=5 os_prio=0 cpu=25.04ms elapsed=1161.40s tid=0x00007feca92711d0 nid=0x22f3f0 in Object.wait()  [0x00007fec5a585000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:170)
    - locked <0x000000061e4602e0> (a java.util.ArrayList)
    at org.testcontainers.utility.ResourceReaper$$Lambda$188/0x0000000800ec8228.run(Unknown Source)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:159)
    at org.testcontainers.utility.ResourceReaper$$Lambda$187/0x0000000800ec8000.run(Unknown Source)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"docker-java-stream-2041008728" #33 daemon prio=5 os_prio=0 cpu=23.42ms elapsed=1160.63s tid=0x00007feca927f250 nid=0x22f564 runnable  [0x00007fec5b488000]
   java.lang.Thread.State: RUNNABLE
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.UnixDomainSocket.recv(Native Method)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.UnixDomainSocket$UnixSocketInputStream.read(UnixDomainSocket.java:246)
    at java.io.FilterInputStream.read(java.base@17.0.1/FilterInputStream.java:132)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.UnixSocketFactory$1$1.read(UnixSocketFactory.java:45)
    at org.testcontainers.shaded.okio.Okio$2.read(Okio.java:140)
    at org.testcontainers.shaded.okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
    at org.testcontainers.shaded.okio.RealBufferedSource.request(RealBufferedSource.java:72)
    at org.testcontainers.shaded.okio.RealBufferedSource.require(RealBufferedSource.java:65)
    at org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:307)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
    at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286)
    at org.testcontainers.shaded.okio.RealBufferedSource$1.read(RealBufferedSource.java:436)
    at org.testcontainers.shaded.com.github.dockerjava.core.FramedInputStreamConsumer.accept(FramedInputStreamConsumer.java:30)
    at org.testcontainers.shaded.com.github.dockerjava.core.FramedInputStreamConsumer.accept(FramedInputStreamConsumer.java:12)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$177/0x0000000800ec0558.run(Unknown Source)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"Attach Listener" #35 daemon prio=9 os_prio=0 cpu=0.60ms elapsed=693.45s tid=0x00007fec54001550 nid=0x22fa58 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=130.82ms elapsed=1164.12s tid=0x00007feca8160df0 nid=0x22f2b4 runnable  

"GC Thread#0" os_prio=0 cpu=30.00ms elapsed=1164.13s tid=0x00007feca805adb0 nid=0x22f2af runnable  

"GC Thread#1" os_prio=0 cpu=27.36ms elapsed=1163.95s tid=0x00007fec680068c0 nid=0x22f2c1 runnable  

"GC Thread#2" os_prio=0 cpu=30.48ms elapsed=1163.95s tid=0x00007fec680072f0 nid=0x22f2c2 runnable  

"GC Thread#3" os_prio=0 cpu=25.00ms elapsed=1163.95s tid=0x00007fec68007d20 nid=0x22f2c3 runnable  

"GC Thread#4" os_prio=0 cpu=32.47ms elapsed=1163.95s tid=0x00007fec68008b40 nid=0x22f2c4 runnable  

"GC Thread#5" os_prio=0 cpu=31.29ms elapsed=1163.95s tid=0x00007fec68009980 nid=0x22f2c5 runnable  

"GC Thread#6" os_prio=0 cpu=28.47ms elapsed=1163.95s tid=0x00007fec6800a7e0 nid=0x22f2c6 runnable  

"GC Thread#7" os_prio=0 cpu=34.28ms elapsed=1163.95s tid=0x00007fec6800b640 nid=0x22f2c7 runnable  

"GC Thread#8" os_prio=0 cpu=35.57ms elapsed=1163.95s tid=0x00007fec6800c4a0 nid=0x22f2c8 runnable  

"GC Thread#9" os_prio=0 cpu=34.38ms elapsed=1163.95s tid=0x00007fec6800d300 nid=0x22f2c9 runnable  

"GC Thread#10" os_prio=0 cpu=30.56ms elapsed=1163.95s tid=0x00007fec6800e160 nid=0x22f2ca runnable  

"GC Thread#11" os_prio=0 cpu=30.47ms elapsed=1162.22s tid=0x00007fec680186d0 nid=0x22f30b runnable  

"GC Thread#12" os_prio=0 cpu=25.04ms elapsed=1162.22s tid=0x00007fec68018d80 nid=0x22f30c runnable  

"G1 Main Marker" os_prio=0 cpu=0.41ms elapsed=1164.13s tid=0x00007feca806b5d0 nid=0x22f2b0 runnable  

"G1 Conc#0" os_prio=0 cpu=3.12ms elapsed=1164.13s tid=0x00007feca806c530 nid=0x22f2b1 runnable  

"G1 Conc#1" os_prio=0 cpu=3.06ms elapsed=1162.53s tid=0x00007fec74000d30 nid=0x22f2da runnable  

"G1 Conc#2" os_prio=0 cpu=1.45ms elapsed=1162.53s tid=0x00007fec74001770 nid=0x22f2dc runnable  

"G1 Refine#0" os_prio=0 cpu=0.10ms elapsed=1164.13s tid=0x00007feca8131e10 nid=0x22f2b2 runnable  

"G1 Service" os_prio=0 cpu=110.02ms elapsed=1164.13s tid=0x00007feca8132d00 nid=0x22f2b3 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=308.21ms elapsed=1163.86s tid=0x00007feca80273a0 nid=0x22f2cc waiting on condition  

JNI global refs: 33, weak refs: 45
evaluateSequential(java.base@17.0.1/ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(java.base@17.0.1/AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(java.base@17.0.1/ReferencePipeline.java:596)
    at org.apache.shardingsphere.test.integration.env.container.atomic.ITContainers.start(ITContainers.java:82)
    - locked <0x000000061cefdc20> (a org.apache.shardingsphere.test.integration.env.container.atomic.ITContainers)
    at org.apache.shardingsphere.test.integration.container.compose.mode.ClusterComposedContainer.start(ClusterComposedContainer.java:64)
    at org.apache.shardingsphere.test.integration.engine.BaseITCase.setUp(BaseITCase.java:78)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.1/Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.1/NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.1/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@17.0.1/Method.java:568)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.73ms elapsed=1164.12s tid=0x00007feca8164e40 nid=0x22f2b5 waiting on condition  [0x00007fec71142000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.ref.Reference.waitForReferencePendingList(java.base@17.0.1/Native Method)
    at java.lang.ref.Reference.processPendingReferences(java.base@17.0.1/Reference.java:253)
    at java.lang.ref.Reference$ReferenceHandler.run(java.base@17.0.1/Reference.java:215)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0.53ms elapsed=1164.12s tid=0x00007feca8166220 nid=0x22f2b6 in Object.wait()  [0x00007fec707fe000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@17.0.1/ReferenceQueue.java:155)
    - locked <0x000000061c5b4578> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(java.base@17.0.1/ReferenceQueue.java:176)
    at java.lang.ref.Finalizer$FinalizerThread.run(java.base@17.0.1/Finalizer.java:172)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.22ms elapsed=1164.12s tid=0x00007feca816b580 nid=0x22f2b7 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0.57ms elapsed=1164.12s tid=0x00007feca816c930 nid=0x22f2b8 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #6 daemon prio=9 os_prio=0 cpu=13.62ms elapsed=1164.12s tid=0x00007feca816dd40 nid=0x22f2b9 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 cpu=1485.55ms elapsed=1164.12s tid=0x00007feca816f770 nid=0x22f2ba waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #15 daemon prio=9 os_prio=0 cpu=489.43ms elapsed=1164.12s tid=0x00007feca8170ca0 nid=0x22f2bb waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #19 daemon prio=9 os_prio=0 cpu=16.61ms elapsed=1164.12s tid=0x00007feca8172110 nid=0x22f2bc runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #20 daemon prio=8 os_prio=0 cpu=1.42ms elapsed=1164.11s tid=0x00007feca817d960 nid=0x22f2bd in Object.wait()  [0x00007fec5b78b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.ref.ReferenceQueue.remove(java.base@17.0.1/ReferenceQueue.java:155)
    - locked <0x000000061c5c00f8> (a java.lang.ref.ReferenceQueue$Lock)
    at jdk.internal.ref.CleanerImpl.run(java.base@17.0.1/CleanerImpl.java:140)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)
    at jdk.internal.misc.InnocuousThread.run(java.base@17.0.1/InnocuousThread.java:162)

"Notification Thread" #22 daemon prio=9 os_prio=0 cpu=0.06ms elapsed=1163.86s tid=0x00007feca8c02320 nid=0x22f2cb runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"surefire-forkedjvm-command-thread" #23 daemon prio=5 os_prio=0 cpu=14.96ms elapsed=1163.80s tid=0x00007feca8c77c40 nid=0x22f2cf runnable  [0x00007fec5a484000]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileInputStream.readBytes(java.base@17.0.1/Native Method)
    at java.io.FileInputStream.read(java.base@17.0.1/FileInputStream.java:276)
    at java.io.BufferedInputStream.fill(java.base@17.0.1/BufferedInputStream.java:244)
    at java.io.BufferedInputStream.read(java.base@17.0.1/BufferedInputStream.java:263)
    - locked <0x000000061c5b9378> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readInt(java.base@17.0.1/DataInputStream.java:393)
    at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
    at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:391)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"surefire-forkedjvm-ping-30s" #24 daemon prio=5 os_prio=0 cpu=746.88ms elapsed=1163.76s tid=0x00007feca8cdfc00 nid=0x22f2d0 waiting on condition  [0x00007fec5a383000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@17.0.1/Native Method)
    - parking to wait for  <0x000000061cc01470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.1/LockSupport.java:252)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@17.0.1/AbstractQueuedSynchronizer.java:1672)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.1/ScheduledThreadPoolExecutor.java:1182)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@17.0.1/ScheduledThreadPoolExecutor.java:899)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.1/ThreadPoolExecutor.java:1062)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1122)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"process reaper" #25 daemon prio=10 os_prio=0 cpu=271.39ms elapsed=1163.74s tid=0x00007febec016eb0 nid=0x22f2d3 waiting on condition  [0x00007fec700f7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@17.0.1/Native Method)
    - parking to wait for  <0x000000061cc02490> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.1/LockSupport.java:252)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@17.0.1/SynchronousQueue.java:401)
    at java.util.concurrent.SynchronousQueue.poll(java.base@17.0.1/SynchronousQueue.java:903)
    at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.1/ThreadPoolExecutor.java:1061)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1122)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"OkHttp ConnectionPool" #28 daemon prio=5 os_prio=0 cpu=34.74ms elapsed=1162.53s tid=0x00007febdc089ad0 nid=0x22f2db in Object.wait()  [0x00007fec58be2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at java.lang.Object.wait(java.base@17.0.1/Object.java:472)
    at org.testcontainers.shaded.okhttp3.internal.connection.RealConnectionPool.lambda$new$0(RealConnectionPool.java:62)
    - locked <0x000000061cd9a6d0> (a org.testcontainers.shaded.okhttp3.internal.connection.RealConnectionPool)
    at org.testcontainers.shaded.okhttp3.internal.connection.RealConnectionPool$$Lambda$129/0x0000000800db4438.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.1/ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.1/ThreadPoolExecutor.java:635)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"testcontainers-ryuk" #31 daemon prio=5 os_prio=0 cpu=25.04ms elapsed=1161.40s tid=0x00007feca92711d0 nid=0x22f3f0 in Object.wait()  [0x00007fec5a585000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(java.base@17.0.1/Native Method)
    - waiting on <no object reference available>
    at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:170)
    - locked <0x000000061e4602e0> (a java.util.ArrayList)
    at org.testcontainers.utility.ResourceReaper$$Lambda$188/0x0000000800ec8228.run(Unknown Source)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:159)
    at org.testcontainers.utility.ResourceReaper$$Lambda$187/0x0000000800ec8000.run(Unknown Source)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"docker-java-stream-2041008728" #33 daemon prio=5 os_prio=0 cpu=23.42ms elapsed=1160.63s tid=0x00007feca927f250 nid=0x22f564 runnable  [0x00007fec5b488000]
   java.lang.Thread.State: RUNNABLE
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.UnixDomainSocket.recv(Native Method)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.UnixDomainSocket$UnixSocketInputStream.read(UnixDomainSocket.java:246)
    at java.io.FilterInputStream.read(java.base@17.0.1/FilterInputStream.java:132)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.UnixSocketFactory$1$1.read(UnixSocketFactory.java:45)
    at org.testcontainers.shaded.okio.Okio$2.read(Okio.java:140)
    at org.testcontainers.shaded.okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
    at org.testcontainers.shaded.okio.RealBufferedSource.request(RealBufferedSource.java:72)
    at org.testcontainers.shaded.okio.RealBufferedSource.require(RealBufferedSource.java:65)
    at org.testcontainers.shaded.okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.java:307)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:492)
### Reason analyze (If you can)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
    at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286)
    at org.testcontainers.shaded.okio.RealBufferedSource$1.read(RealBufferedSource.java:436)
    at org.testcontainers.shaded.com.github.dockerjava.core.FramedInputStreamConsumer.accept(FramedInputStreamConsumer.java:30)
    at org.testcontainers.shaded.com.github.dockerjava.core.FramedInputStreamConsumer.accept(FramedInputStreamConsumer.java:12)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:275)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$177/0x0000000800ec0558.run(Unknown Source)
    at java.lang.Thread.run(java.base@17.0.1/Thread.java:833)

"Attach Listener" #35 daemon prio=9 os_prio=0 cpu=0.60ms elapsed=693.45s tid=0x00007fec54001550 nid=0x22fa58 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=130.82ms elapsed=1164.12s tid=0x00007feca8160df0 nid=0x22f2b4 runnable  

"GC Thread#0" os_prio=0 cpu=30.00ms elapsed=1164.13s tid=0x00007feca805adb0 nid=0x22f2af runnable  

"GC Thread#1" os_prio=0 cpu=27.36ms elapsed=1163.95s tid=0x00007fec680068c0 nid=0x22f2c1 runnable  

"GC Thread#2" os_prio=0 cpu=30.48ms elapsed=1163.95s tid=0x00007fec680072f0 nid=0x22f2c2 runnable  

"GC Thread#3" os_prio=0 cpu=25.00ms elapsed=1163.95s tid=0x00007fec68007d20 nid=0x22f2c3 runnable  

"GC Thread#4" os_prio=0 cpu=32.47ms elapsed=1163.95s tid=0x00007fec68008b40 nid=0x22f2c4 runnable  

"GC Thread#5" os_prio=0 cpu=31.29ms elapsed=1163.95s tid=0x00007fec68009980 nid=0x22f2c5 runnable  

"GC Thread#6" os_prio=0 cpu=28.47ms elapsed=1163.95s tid=0x00007fec6800a7e0 nid=0x22f2c6 runnable  

"GC Thread#7" os_prio=0 cpu=34.28ms elapsed=1163.95s tid=0x00007fec6800b640 nid=0x22f2c7 runnable  

"GC Thread#8" os_prio=0 cpu=35.57ms elapsed=1163.95s tid=0x00007fec6800c4a0 nid=0x22f2c8 runnable  

"GC Thread#9" os_prio=0 cpu=34.38ms elapsed=1163.95s tid=0x00007fec6800d300 nid=0x22f2c9 runnable  

"GC Thread#10" os_prio=0 cpu=30.56ms elapsed=1163.95s tid=0x00007fec6800e160 nid=0x22f2ca runnable  

"GC Thread#11" os_prio=0 cpu=30.47ms elapsed=1162.22s tid=0x00007fec680186d0 nid=0x22f30b runnable  

"GC Thread#12" os_prio=0 cpu=25.04ms elapsed=1162.22s tid=0x00007fec68018d80 nid=0x22f30c runnable  

"G1 Main Marker" os_prio=0 cpu=0.41ms elapsed=1164.13s tid=0x00007feca806b5d0 nid=0x22f2b0 runnable  

"G1 Conc#0" os_prio=0 cpu=3.12ms elapsed=1164.13s tid=0x00007feca806c530 nid=0x22f2b1 runnable  

"G1 Conc#1" os_prio=0 cpu=3.06ms elapsed=1162.53s tid=0x00007fec74000d30 nid=0x22f2da runnable  

"G1 Conc#2" os_prio=0 cpu=1.45ms elapsed=1162.53s tid=0x00007fec74001770 nid=0x22f2dc runnable  

"G1 Refine#0" os_prio=0 cpu=0.10ms elapsed=1164.13s tid=0x00007feca8131e10 nid=0x22f2b2 runnable  

"G1 Service" os_prio=0 cpu=110.02ms elapsed=1164.13s tid=0x00007feca8132d00 nid=0x22f2b3 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=308.21ms elapsed=1163.86s tid=0x00007feca80273a0 nid=0x22f2cc waiting on condition  

JNI global refs: 33, weak refs: 45

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

The scenario may be changed.

             ./mvnw -B install -f shardingsphere-test/shardingsphere-integration-test/shardingsphere-integration-test-suite/pom.xml \
                 -Dit.run.modes=Cluster \
                 -Dit.cluster.databases=MySQL \
                 -Dit.cluster.adapters=proxy \
                 -Dit.scenarios=dbtbl_with_readwrite_splitting \
                 -Dit.cluster.env.type=DOCKER
TeslaCN commented 2 years ago

It seems that the thread is stuck in a dead loop.

image

image

I don't know if this is normal that the pos is greater than limit. image

TeslaCN commented 2 years ago

https://github.com/square/okio/issues/1133

github-actions[bot] commented 2 years ago

Hello , this issue has not received a reply for several days. This issue is supposed to be closed.

TeslaCN commented 2 years ago

This issue hasn't happened recently.