Can produce output yet will have some other errors

[alvin9964]

2020-09-09 14:01:37 WARN BatchNormalization:382 - CuDNN BatchNormalization forward pass execution failed - falling back on built-in implementation java.lang.RuntimeException: cuDNN status = 3: CUDNN_STATUS_BAD_PARAM at org.deeplearning4j.cuda.BaseCudnnHelper.checkCudnn(BaseCudnnHelper.java:48) at org.deeplearning4j.cuda.normalization.CudnnBatchNormalizationHelper.preOutput(CudnnBatchNormalizationHelper.java:320) at org.deeplearning4j.nn.layers.normalization.BatchNormalization.preOutput(BatchNormalization.java:462) at org.deeplearning4j.nn.layers.normalization.BatchNormalization.activate(BatchNormalization.java:404) at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:111) at org.deeplearning4j.nn.graph.ComputationGraph.outputOfLayersDetached(ComputationGraph.java:2380) at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1741) at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1697) at org.deeplearning4j.nn.graph.ComputationGraph.output(ComputationGraph.java:1627) at SocialDistanceCheckerVideo.getPredictedObjects(SocialDistanceCheckerVideo.java:150) at SocialDistanceCheckerVideo.main(SocialDistanceCheckerVideo.java:76)

Pom file as below : <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

4.0.0

<groupId>Certifai</groupId>
<artifactId>SocialDistancingChecker</artifactId>
<version>1.0-SNAPSHOT</version>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>8</source>
                <target>8</target>
            </configuration>
        </plugin>
    </plugins>
</build>
<properties>
    <nd4j.backend>nd4j-native-platform</nd4j.backend>
    <nd4j.backend>nd4j-cuda-10.2-platform</nd4j.backend>
    <nd4j.version>1.0.0-beta7</nd4j.version>
    <dl4j.version>1.0.0-beta7</dl4j.version>
    <datavec.version>1.0.0-beta7</datavec.version>
</properties>

<repositories>
    <repository>
        <id>snapshots-repo</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>org.nd4j</groupId>
        <artifactId>${nd4j.backend}</artifactId>
        <version>${nd4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-cuda-10.2</artifactId>
        <version>${dl4j.version}</version>
    </dependency>
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacv-platform</artifactId>
        <version>1.5.3</version>
    </dependency>
    <dependency>
        <groupId>org.deeplearning4j</groupId>
        <artifactId>deeplearning4j-ui</artifactId>
        <version>${dl4j.version}</version>
    </dependency>
    <!--        Logging libraries-->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>2.0.0-alpha1</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>2.0.0-alpha1</version>
    </dependency>
</dependencies>

Cuda version 10.2 Cudnn 7.6.5

[bewithme]

can you reproduce this problem ?

[alvin9964]

So apparently I tired several combination of CUDA version with CUDNN version, here is the result

10.1 - 7.6.1 same error 10.1 - 7.6.2 OK! 10.1 - 7.6.3 OK! 10.1 - 7.6.4 OK! 10.1 - 7.6.5 OK!

10.2 - 7.6.5 same error

I think the workaround solution for me is to prevent those CUDNN versions that cause error and stick with CUDA 10.1 first.