OpenBLAS binary not found

[b005t3r]

I've run into issues with my app which has OpenBLAS added as a dependency (via deeplearning4j and OpenCV, I think, not sure if it is even necessary).

When the app is started, I get this:

Exception in thread "AWT-EventQueue-0" java.lang.UnsatisfiedLinkError: /Users/booster/Library/Developer/Xcode/DerivedData/ShotTracerPro-cvzekkwtbhdgfpeynyblsqmazats/Build/Products/Debug/ShotTracerPro.app/Contents/Resources/mactracer/nativeLib/libjniopenblas_nolapack.dylib: dlopen(/Users/booster/Library/Developer/Xcode/DerivedData/ShotTracerPro-cvzekkwtbhdgfpeynyblsqmazats/Build/Products/Debug/ShotTracerPro.app/Contents/Resources/mactracer/nativeLib/libjniopenblas_nolapack.dylib, 1): Library not loaded: @rpath/libopenblas_nolapack.0.dylib
  Referenced from: /Users/booster/Library/Developer/Xcode/DerivedData/ShotTracerPro-cvzekkwtbhdgfpeynyblsqmazats/Build/Products/Debug/ShotTracerPro.app/Contents/Resources/mactracer/nativeLib/libjniopenblas_nolapack.dylib

It looks like it can't load the libopenblas_nolapack.0.dylib which is correct, because there is no such file, however there is a jni lib created for that file (libjniopenblas_nolapack.dylib).

The only caveat is all of that happens when I build the whole project as a single jar file (mvn clean compile assembly:single -U), move all of the libs to a separate directory (they need to be in a separate dir, so they can be signed when creating a build for the Mac App Store) and run the app with '-Djava.library.path=libDir. This is the first time I had issues with this however, everything else loads up just fine, so I'm wondering what might be special about this particular lib. Any ideas on how I can fix it?

Here's a list of my project's dependencies:

        <dependency>
            <groupId>org.tensorflow</groupId>
            <artifactId>libtensorflow</artifactId>
            <version>1.15.0</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.tensorflow</groupId>
            <artifactId>libtensorflow_jni</artifactId>
            <version>1.15.0</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <version>1.0.0-beta7</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native</artifactId>
            <version>1.0.0-beta7</version>
            <classifier>macosx-x86_64-avx2</classifier>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>1.0.0-beta7</version>
            <scope>compile</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.datavec</groupId>
                    <artifactId>datavec-data-image</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.1.7</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacv</artifactId>
            <version>1.5.3</version>
            <scope>compile</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>flycapture</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>libdc1394</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>libfreenect</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>libfreenect2</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>librealsense</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>librealsense2</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>videoinput</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>artoolkitplus</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>flandmark</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>leptonica</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bytedeco</groupId>
                    <artifactId>tesseract</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>openblas</artifactId>
            <version>0.3.9-1.5.3</version>
            <scope>compile</scope>
            <classifier>macosx-x86_64</classifier>
        </dependency>

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>ffmpeg</artifactId>
            <version>4.2.2-1.5.3</version>
            <scope>compile</scope>
            <classifier>macosx-x86_64</classifier>
        </dependency>

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>opencv</artifactId>
            <version>4.3.0-1.5.3</version>
            <scope>compile</scope>
            <classifier>macosx-x86_64</classifier>
        </dependency>

[saudet]

That's an issue with old versions of Spring Boot. Please update to 1.5.4.

[b005t3r]

Can't update, deeplearning4j has the version 1.5.3 set as a dependency, so it'll get used anyway :/

However, I found a simple workaround with creating a symlink to the existing libopenblas.0.dylib and it looks like this is working just fine. However, I still don't understand how on earth did this work in a standard build.

[saudet]

Of course we can override it with 1.5.4. That's what those dependencies are for.

[b005t3r]

OK, how do I do that then? :)

[saudet]

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacpp</artifactId>
            <version>1.5.4</version>
        </dependency>

[saudet]

If you're saying that your app can't use a cache for executables because of some policy from Apple, you'll probably need to create that symbolic link manually yes.

[b005t3r]

No, I mean, how do I change the version of javacpp/javacv that another project wants to use, if that's even possible.

org.nd4j:nd4j-native:jar:1.0.0-beta7:compile has openblas 1.5.3 set as a dependency:

+- org.nd4j:nd4j-native:jar:1.0.0-beta7:compile [INFO] | +- org.nd4j:nd4j-native:jar:macosx-x86_64:1.0.0-beta7:compile [INFO] | | +- (org.nd4j:nd4j-native:jar:macosx-x86_64:1.0.0-beta7:compile - omitted for cycle) [INFO] | | +- (org.nd4j:nd4j-native:jar:macosx-x86_64:1.0.0-beta7:compile - omitted for duplicate) [INFO] | | +- (org.bytedeco:javacpp:jar:1.5.3:compile - omitted for duplicate) [INFO] | | +- (org.bytedeco:javacpp:jar:macosx-x86_64:1.5.3:compile - omitted for duplicate) [INFO] | | +- (org.bytedeco:openblas:jar:0.3.9-1-1.5.3:compile - omitted for duplicate) [INFO] | | +- (org.bytedeco:openblas:jar:macosx-x86_64:0.3.9-1-1.5.3:compile - omitted for duplicate) [INFO] | | +- (org.nd4j:nd4j-native-api:jar:1.0.0-beta7:compile - omitted for duplicate) [INFO] | | +- (org.nd4j:nd4j-api:jar:1.0.0-beta7:compile - omitted for duplicate) [INFO] | | +- (org.bytedeco:mkl:jar:2020.1-1.5.3:compile - omitted for duplicate) [INFO] | | - (org.bytedeco:mkl:jar:macosx-x86_64:2020.1-1.5.3:compile - omitted for duplicate) [INFO] | +- org.bytedeco:javacpp:jar:1.5.3:compile [INFO] | +- org.bytedeco:javacpp:jar:macosx-x86_64:1.5.3:compile [INFO] | +- org.bytedeco:openblas:jar:0.3.9-1-1.5.3:compile

That's why I used the same version for my project.

[saudet]

Sure, you can override those versions as well. That's what Maven is for: https://maven.apache.org/plugins/maven-dependency-plugin/examples/resolving-conflicts-using-the-dependency-tree.html

[b005t3r]

Hah, OK, I'll give it a go :) So, to be clear, with 1.5.4 I no longer need OpenBLAS or the OpenBLAS built for this version no longer has the missing binary issue?

[saudet]

It's not "missing", it still needs to be able to create a symbolic link.

[b005t3r]

OK, so I would still need to create a symlink for my build? What's the gain then?

[saudet]

It might not change anything. Let me know if you give it a try though.

[b005t3r]

I like it how we often have those small conversations on how not to change anything :)