search

testcontainers / testcontainers-java

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
https://testcontainers.org
MIT License
8.03k stars 1.65k forks source link

Cannot use testcontainers with Confluent Kafka 6.0.0 #3288

Closed maver1ck closed 1 year ago

maver1ck commented 4 years ago

Hi, I tried to create Confluent Kafka 6.0.0 container. As a result I'm getting exception:

11:13:52.846 [testcontainers-lifecycle-0] ERROR 🐳 [confluentinc/cp-kafka:6.0.0] - Could not start container
java.lang.IllegalStateException: Container exited with code 126
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:470) ~[testcontainers-1.14.3.jar:?]
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325) ~[testcontainers-1.14.3.jar:?]
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) ~[duct-tape-1.0.8.jar:?]
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323) ~[testcontainers-1.14.3.jar:?]
    at org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:102) ~[kafka-1.14.3.jar:?]
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311) ~[testcontainers-1.14.3.jar:?]
    at java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:783) [?:?]
    at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
11:13:52.889 [testcontainers-lifecycle-0] ERROR 🐳 [confluentinc/cp-kafka:6.0.0] - Log output from the failed container:
sh: /testcontainers_start.sh: Permission denied
gAmUssA commented 4 years ago

@maver1ck kindly provide steps to reproduce. Thanks

maver1ck commented 4 years ago

@gAmUssA

Package versions:

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.7.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>kafka</artifactId>
            <version>1.14.3</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>1.14.3</version>
            <scope>test</scope>
        </dependency>

Code:

import org.junit.jupiter.api.Test;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import static org.junit.jupiter.api.Assertions.assertTrue;

@Testcontainers
public class FailKafkaTest {

    @Container
    private static final KafkaContainer kafka = new KafkaContainer("6.0.0");

    @Test
    public void testKafkaIsRunning() {
        assertTrue(kafka.isRunning());
    }
}

Full test logs:

14:19:08.424 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.WindowsClientProviderStrategy from ~/.testcontainers.properties, will try it first
14:19:09.665 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Docker for Windows (via TCP port 2375)
14:19:10.240 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is localhost
14:19:10.309 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 19.03.12
  API Version: 1.40
  Operating System: Ubuntu 20.04.1 LTS
  Total Memory: 7960 MB
14:19:11.154 [main] WARN  org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: testcontainersofficial/ryuk:0.3.0, configFile: C:\Users\maverick\.docker\config.json. Falling back to docker-java default behaviour. Exception message: C:\Users\maverick\.docker\config.json (The system cannot find the path specified)
14:19:13.599 [main] INFO  org.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
14:19:13.600 [main] INFO  org.testcontainers.DockerClientFactory - Checking the system...
14:19:13.602 [main] INFO  org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
14:19:14.357 [main] INFO  org.testcontainers.DockerClientFactory - ✔︎ Docker environment should have more than 2GB free disk space
14:19:14.509 [main] INFO  🐳 [confluentinc/cp-kafka:6.0.0] - Creating container for image: confluentinc/cp-kafka:6.0.0
14:19:14.747 [main] INFO  🐳 [confluentinc/cp-kafka:6.0.0] - Starting container with ID: 1a0db2bfb6c615b9242bd977981a15ed5f466c131f63cccccbc374ac76314219
14:19:15.304 [main] INFO  🐳 [confluentinc/cp-kafka:6.0.0] - Container confluentinc/cp-kafka:6.0.0 is starting: 1a0db2bfb6c615b9242bd977981a15ed5f466c131f63cccccbc374ac76314219
14:20:15.575 [main] ERROR 🐳 [confluentinc/cp-kafka:6.0.0] - Could not start container
java.lang.IllegalStateException: Container exited with code 126
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:470) ~[testcontainers-1.14.3.jar:?]
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325) ~[testcontainers-1.14.3.jar:?]
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) ~[duct-tape-1.0.8.jar:?]
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323) ~[testcontainers-1.14.3.jar:?]
    at org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:102) ~[kafka-1.14.3.jar:?]
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311) ~[testcontainers-1.14.3.jar:?]
    at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242) ~[junit-jupiter-1.14.3.jar:?]
    at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229) ~[junit-jupiter-1.14.3.jar:?]
    at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59) ~[junit-jupiter-1.14.3.jar:?]
    at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$4(ExtensionValuesStore.java:86) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:205) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.evaluate(ExtensionValuesStore.java:182) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.access$100(ExtensionValuesStore.java:171) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:89) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59) ~[junit-jupiter-1.14.3.jar:?]
    at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
    at org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59) ~[junit-jupiter-1.14.3.jar:?]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$8(ClassBasedTestDescriptor.java:368) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:368) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:192) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78) ~[junit-jupiter-engine-5.7.0.jar:5.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:136) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51) ~[junit-platform-engine-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108) ~[junit-platform-launcher-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) ~[junit-platform-launcher-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) ~[junit-platform-launcher-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) [junit-platform-launcher-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) [junit-platform-launcher-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96) [junit-platform-launcher-1.7.0.jar:1.7.0]
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75) [junit-platform-launcher-1.7.0.jar:1.7.0]
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71) [junit5-rt.jar:?]
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) [junit-rt.jar:?]
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220) [junit-rt.jar:?]
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53) [junit-rt.jar:?]
14:20:15.633 [main] ERROR 🐳 [confluentinc/cp-kafka:6.0.0] - Log output from the failed container:
sh: /testcontainers_start.sh: Permission denied

With Kafka 5.5.1 everything is working properly.

14:21:14.043 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.WindowsClientProviderStrategy from ~/.testcontainers.properties, will try it first
14:21:15.143 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Docker for Windows (via TCP port 2375)
14:21:15.401 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is localhost
14:21:15.440 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 19.03.12
  API Version: 1.40
  Operating System: Ubuntu 20.04.1 LTS
  Total Memory: 7960 MB
14:21:15.580 [main] WARN  org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: testcontainersofficial/ryuk:0.3.0, configFile: C:\Users\maverick\.docker\config.json. Falling back to docker-java default behaviour. Exception message: C:\Users\maverick\.docker\config.json (The system cannot find the path specified)
14:21:16.992 [main] INFO  org.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
14:21:16.993 [main] INFO  org.testcontainers.DockerClientFactory - Checking the system...
14:21:16.994 [main] INFO  org.testcontainers.DockerClientFactory - ✔︎ Docker server version should be at least 1.6.0
14:21:17.134 [main] INFO  org.testcontainers.DockerClientFactory - ✔︎ Docker environment should have more than 2GB free disk space
14:21:17.427 [main] INFO  🐳 [confluentinc/cp-kafka:5.5.1] - Creating container for image: confluentinc/cp-kafka:5.5.1
14:21:17.898 [main] INFO  🐳 [confluentinc/cp-kafka:5.5.1] - Starting container with ID: acfa040fa958101075938905965ae530b8123b861389ed40e2d2888884bdc569
14:21:18.521 [main] INFO  🐳 [confluentinc/cp-kafka:5.5.1] - Container confluentinc/cp-kafka:5.5.1 is starting: acfa040fa958101075938905965ae530b8123b861389ed40e2d2888884bdc569
14:21:28.698 [main] INFO  🐳 [confluentinc/cp-kafka:5.5.1] - Container confluentinc/cp-kafka:5.5.1 started in PT14.7348488S

Process finished with exit code 0
bsideup commented 4 years ago

Submitted https://github.com/testcontainers/testcontainers-java/pull/3293

astubbs commented 3 years ago

Same issue, CP 6.0.1.

Reproduced: https://github.com/confluentinc/parallel-consumer/pull/72 (commit: https://github.com/confluentinc/parallel-consumer/pull/72/commits/30be119b85b3fba1f673c50b84f6b9a380ba4965)

16:15:28  15:26.870  ERROR [main]  (GenericContainer.java:492#tryStart) Log output from the failed container:
16:15:28  sh: /testcontainers_start.sh: Permission denied

@gAmUssA msg me if you want more info or want to iterate. Happens with 6.0.0 as well. I found it as I was using :latest, had to switch to 5.5.3 which works fine it seems.

teabot commented 3 years ago

I'm finding this with 6.1.1 also

LeonardoBonacci commented 3 years ago

6.2.0 too

kwart commented 1 year ago

We've hit this issue (or a very similar one) with confluentinc/cp-kafka:7.3.0 and org.testcontainers:kafka:jar:1.18.0.

Here is the test log:

21:45:39,072  INFO || - [0] Time-limited test - Creating container for image: confluentinc/cp-kafka:7.3.0
21:45:40,269  INFO || - [0] Time-limited test - Container confluentinc/cp-kafka:7.3.0 is starting: 89592968150f0d1da6245c7751059be6bb2a12c08a413a306404887678d04fbc
21:45:41,223  INFO || - [DockerizedKafkaTestSupport] docker-java-stream-2127774505 - STDERR: sh: /testcontainers_start.sh: /bin/bash: bad interpreter: Text file busy
21:46:41,247 ERROR || - [0] Time-limited test - Could not start container
java.lang.IllegalStateException: Wait strategy failed. Container exited with code 126
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:523) ~[testcontainers-1.18.0.jar:?]
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344) ~[testcontainers-1.18.0.jar:?]
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81) [duct-tape-1.0.8.jar:?]
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334) [testcontainers-1.18.0.jar:?]
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322) [testcontainers-1.18.0.jar:?]
    at com.hazelcast.jet.kafka.impl.DockerizedKafkaTestSupport.createKafkaCluster(DockerizedKafkaTestSupport.java:40) [test-classes/:?]
    at com.hazelcast.jet.kafka.impl.StreamKafkaPTest.beforeClass(StreamKafkaPTest.java:107) [test-classes/:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) [junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.13.2.jar:4.13.2]
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) [junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) [junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) [junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.13.2.jar:4.13.2]
    at com.hazelcast.test.AfterClassesStatement.evaluate(AfterClassesStatement.java:41) [hazelcast-5.3.0-SNAPSHOT-tests.jar:5.3.0-SNAPSHOT]
    at com.hazelcast.test.OverridePropertyRule$1.evaluate(OverridePropertyRule.java:66) [hazelcast-5.3.0-SNAPSHOT-tests.jar:5.3.0-SNAPSHOT]
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) [junit-4.13.2.jar:4.13.2]
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) [junit-4.13.2.jar:4.13.2]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*\[KafkaServer id=\d+\] started.*'
    at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47) ~[testcontainers-1.18.0.jar:1.18.0]
    at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52) ~[testcontainers-1.18.0.jar:1.18.0]
    at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:964) ~[testcontainers-1.18.0.jar:?]
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:490) ~[testcontainers-1.18.0.jar:?]
    ... 22 more
21:46:41,265 ERROR || - [0] Time-limited test - Log output from the failed container:
sh: /testcontainers_start.sh: /bin/bash: bad interpreter: Text file busy
kwart commented 1 year ago

Probably the root of the evil is a race-condition between the container's CMD waiting for the startup-script (here) and copying the script itself from the outside of the container (here).

maver1ck commented 1 year ago

I was able to run Confluent Kafka 7.5.1 so closing this.