[Java 16] cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module

[blueberry]

The --add-opens option does not seem to work with Java 16 for the DirectByteBuffer cleaner, which is important part of (release whatever-native-object).

The solution is to use jdk.internal.misc.Unsafe and -add-opens "--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED" (note different module, misc instead of ref).

However, this will not work on Java 8, so there is hard choice between:

• Supporting Java 16 (which should happen eventually)
• Still supporting Java 8 (which is still somewhat relevant)

I'm inclined to let Java 8 work for now, and when enough people ask for Java 16 support, abandon Java 8 in new releases. So, please voice your opinion here (or suggest something else).

[blueberry]

UPDATE: The following is not a solution!!!!

Here is how the solution is implemented in NEW (Java 16 compatible) uncomplicate/commons:

(extend-type ByteBuffer
  Releaseable
  (release [this]
    (.invokeCleaner (jdk.internal.misc.Unsafe/getUnsafe) this)
    true))

If you need it right now, you can evaluate that code in your project, and this should work (of course, import ByteBuffer and require Releaseable before that).

Don't forget to ADD NEW :jvm-options "--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED" (see https://github.com/uncomplicate/commons/blob/master/project.clj)

[blueberry]

There's some shenanigans either with how Java 16 works, or how maven/leiningen transitive dependencies work with Leiningen.

Old way works well with Java 16 in uncomplicate/commons, but then when called from neanderthal, the same module opens do not work. This need more investigation. For now, please use Java 15 or lower...

[blueberry]

OK, there seem to be some shenanigans, BUT, at the end, this should be the solution:

"--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"

At first it did not work for me, in neanderthal tests, for some reason, but then I started trying out different things with parts of text of project.clj (unrelated to this), and the same text, but written/copied from elsewhere worked. The same old text didn't...

So, in your project, it should be sufficient to just use the additional --add-open that I mentioned...

[jsa-aerial]

Hi Dragan - are you saying the above works for both Java 8 and 16? Thanks.

[blueberry]

Yes! Comment for Java 8, uncomment for Java 9+ (including Java 16).

[BorisVSchmid]

For those who come behind. You can get the error of "cannot access class .. sun.nio.ch" also when doing something minor wrong like feeding two vectors of different lengths to fluokitten fmap!.

(uncomplicate.fluokitten.core/fmap! (fn [a b] (if (> a b) -0.5 0.5)) (fv 1 2 3 4) (fv 4 3 2 1))
  #RealBlockVector[float, n:4, offset: 0, stride:1]
  [   0.50    0.50   -0.50   -0.50 ]

  (uncomplicate.fluokitten.core/fmap! (fn [a b] (if (> a b) -0.5 0.5)) (fv 1 2 3 4 5) (fv 4 3 2 1))
  ; Execution error (IllegalAccessError) at uncomplicate.commons.core/eval6181$fn (core.clj:145).
  ; class uncomplicate.commons.core$eval6181$fn__6182 (in unnamed module @0x3765e155) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x3765e155  

This is on java 17.0.1 / clojure 1.11.1 / and

uncomplicate/commons {:mvn/version "0.13.0"}
          uncomplicate/fluokitten {:mvn/version "0.9.1"}
          uncomplicate/neanderthal {:mvn/version "0.45.0"}
          org.jcuda/jcudnn {:mvn/version "11.7.0"}
          org.bytedeco/mkl-platform-redist {:mvn/version "2020.3-1.5.4"}

[blueberry]

Hi Boris, I'm getting a different exception, which is expected: " uncomplicate.neanderthal.internal.host.buffer_block.RealBlockVector cannot be cast to class java.lang.Number"

I'm not on Java 17, but OpenJDK 18...

Does it only happen on Java 17, or you have this problem on other setups?

[BorisVSchmid]

Looks like the problem was a misalignment between what version of jcublas, what version of Neanderthal & what version of NVIDIA cuda toolkit was installed.

When I made sure the deps.edn (or project.clj) has the dependency versions as listed on clojars, sanity was restored (at least for java 8, haven't yet tried the higher versions)

https://clojars.org/uncomplicate/neanderthal/versions/0.44.1

So, a good first attempt at debugging these "cannot access class" errors is to check if your cuda toolkit, jcublas version and neanderthal version are still aligned.

[BorisVSchmid]

Found what was the root cause of the problem was in my case:

I was using clojure CLI tools, and had a deps.edn running rather than a project.clj. The mistake I made was having a jvm-opts as a top-level keyword in the deps.edn. Those get ignored - jvm-opts have to be defined within an alias. See below.

{:description "If it works with lein repl / (load-file), it should work with clj / deps.tools"

 :paths ["src"]

 :deps {org.clojure/clojure {:mvn/version "1.10.3"}
        uncomplicate/neanderthal {:mvn/version "0.45.0"}
        org.bytedeco/mkl-platform-redist {:mvn/version "2020.3-1.5.4"}}

 :exclusions [[org.jcuda/jcuda-natives :classifier "apple-x86_64"]
              [org.jcuda/jcublas-natives :classifier "apple-x86_64"]]

 :aliases {:develop 
           {:jvm-opts ["-Dclojure.compiler.direct-linking=true"
                       "-XX:MaxDirectMemorySize=16g" "-XX:+UseLargePages"
                       "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"
                       "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"]}}}

;; jvm-opts have to be within an alias, and not in the top level of the map.
;; get into the repl with "clj -A:develop", and enter (load-file "src/hello_world/cuda.clj") to execute the gpu test.

And in Calva I got into trouble by jacking in and connecting my project without selecting any aliases. The repl within calva will run fine when just pressing enter, but you are running without jvm-opts then!

[Yuvraj2172]

Can someone provide the same solution for build.sbt file ?

[ff137]

Can someone provide the same solution for build.sbt file ? @Yuvraj2172

lazy val root= (project in file("."))
  .settings(
  ...,
    javaOptions ++= Seq( // Spark-specific JVM options
      "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
    )
  )

[blueberry]

Do you still need one of the earlier versions? From 0.48.0 forward, you do not need to mess with specific options. (please see https://github.com/uncomplicate/neanderthal/blob/master/examples/hello-world/project.clj)

[ff137]

just got the same issue from using Spark. Came from google

[blueberry]

Which Neanderthal version are we talking about?

[ff137]

Soz I meant I'm not using this project, just wanted to tell the guy how to set Java option in build SBT 😂