Call methods of a class generated in runtime

[dfdx]

jcall fails with NoSuchMethodError when calling methods of a dynamically compiled class.

I use InMemoryJavaCompiler to compile a new class and create its instance. (InMemoryJavaCompiler is based on javax.tools.*, so quite standard approach).

```julia const JInMemoryJavaCompiler = @jimport org.mdkt.compiler.InMemoryJavaCompiler function mkclass(name::String, src::String) jcompiler = jcall(JInMemoryJavaCompiler, "newInstance", JInMemoryJavaCompiler, ()) return jcall(jcompiler, "compile", JClass, (JString, JString), name, src) end function instantiate(name::String, src::String) jclass = mkclass(name, src) return jcall(jclass, "newInstance", JObject, ()) end init() name = "julia.compiled.Hello" src = """ package julia.compiled; public class Hello { public String hello(String name) { return "Hello, " + name; } public void goodbye() { System.out.println("Goodbye!"); } } """ obj = instantiate(name, src) ```

The resulting object looks like the ordinary one and I can see the methods I defined:

julia> listmethods(obj)
11-element Vector{JMethod}:
 java.lang.String hello(java.lang.String)
 void goodbye()
 void wait(long)
 void wait(long, int)
 void wait()
 boolean equals(java.lang.Object)
 java.lang.String toString()
 int hashCode()
 java.lang.Class getClass()
 void notify()
 void notifyAll()

However, calling them with jcall fails, even though calling the inherited methods works fine:

```julia julia> jcall(obj, "hello", JString, (JString,), "Bob") Exception in thread "main" java.lang.NoSuchMethodError: hello ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodError: hello") Stacktrace: [1] geterror(allow::Bool) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:418 [2] jcall(obj::JObject, method::String, rettype::Type, argtypes::Tuple{DataType}, args::String) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:244 [3] top-level scope @ REPL[13]:1 julia> jcall(obj, "goodbye", Nothing, ()) Exception in thread "main" java.lang.NoSuchMethodError: goodbye ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodError: goodbye") Stacktrace: [1] geterror(allow::Bool) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:418 [2] jcall(::JObject, ::String, ::Type, ::Tuple{}) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:244 [3] top-level scope @ REPL[14]:1 julia> jcall(obj, "equals", jboolean, (JObject,), obj) 0x01 ```

What's even more confusing, these new methods can actually be called using the reflection:

```julia function jcall2(jobj::JavaObject, name::String, ret_type, arg_types, args...) jclass = getclass(jobj) jargs = [a for a in convert.(arg_types, args)] # convert to Vector meth = jcall(jclass, "getMethod", JMethod, (JString, Vector{JClass}), name, getclass.(jargs)) return meth(jobj, jargs...) end julia> jcall2(obj, "hello", JString, (JString,), "Bob") "Hello, Bob" julia> jcall2(obj, "goodbye", Nothing, ()) Goodbye! ```

Although it fails with NoSuchMethodException for the inherited method ¯_(ツ)_/¯:

```julia julia> jcall2(obj, "equals", jboolean, (JObject,), obj) Exception in thread "main" java.lang.NoSuchMethodException: julia.compiled.Hello.equals(julia.compiled.Hello) at java.base/java.lang.Class.getMethod(Class.java:2108) ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodException: julia.compiled.Hello.equals(julia.compiled.Hello)") Stacktrace: [1] geterror(allow::Bool) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:418 [2] geterror @ ~/.julia/packages/JavaCall/MlduK/src/core.jl:403 [inlined] [3] _jcall(::JClass, ::Ptr{Nothing}, ::Ptr{Nothing}, ::Type, ::Tuple{DataType, DataType}, ::String, ::Vararg{Any}) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:373 [4] jcall(::JClass, ::String, ::Type, ::Tuple{DataType, DataType}, ::String, ::Vararg{Any}) @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:245 [5] jcall2(jobj::JObject, name::String, ret_type::Type, arg_types::Tuple{DataType}, args::JObject) @ Main ./REPL[17]:4 [6] top-level scope @ REPL[18]:1 ```

The code above is executable from the Spark#new-api branch, but honestly I hope that I'm just doing some stupid mistake that a keen eye can catch just from the description.

[mkitti]

Could you clarify which version of JavaCall.jl are using?

Note that there is https://github.com/JuliaInterop/JavaCall.jl/releases/tag/v0.8.0rc-1

[dfdx]

I tested it on v0.7.8, but v0.8.0rc-1 gives the same result anyway:

julia> jcall(obj, "hello", JString, (JString,), "Bob")
Exception in thread "main" java.lang.NoSuchMethodError: hello
ERROR: JavaCall.JavaCallError("Error calling Java: java.lang.NoSuchMethodError: hello")
Stacktrace:
 [1] geterror()
   @ JavaCall ~/.julia/packages/JavaCall/KFY5m/src/core.jl:542
 [2] get_method_id(jnifun::typeof(JavaCall.JNI.GetMethodID), obj::JObject, method::String, rettype::Type, argtypes::Tuple{DataType})
   @ JavaCall ~/.julia/packages/JavaCall/KFY5m/src/core.jl:255
 [3] get_method_id
   @ ~/.julia/packages/JavaCall/KFY5m/src/core.jl:393 [inlined]
 [4] jcall(ref::JObject, method::String, rettype::Type, argtypes::Tuple{DataType}, args::String)
   @ JavaCall ~/.julia/packages/JavaCall/KFY5m/src/core.jl:370
 [5] top-level scope
   @ REPL[9]:1

And just in case:

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)
Environment:
  JULIA_COPY_STACKS = 1
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 

[mkitti]

I notice that they use a dynamic class loader. Do they mention anything regarding the Java Native Interface?

[dfdx]

Not really. I will try it with the standard ClassLoader.

[dfdx]

Nope, the same error.

Self-containing example

```julia using JavaCall const JFile = @jimport java.io.File const JToolProvider = @jimport javax.tools.ToolProvider const JJavaCompiler = @jimport javax.tools.JavaCompiler const JInputStream = @jimport java.io.InputStream const JOutputStream = @jimport java.io.OutputStream const JClassLoader = @jimport java.lang.ClassLoader const JURLClassLoader = @jimport java.net.URLClassLoader const JURI = @jimport java.net.URI const JURL = @jimport java.net.URL function JavaCall.classforname(name::String, loader) return jcall(JClass, "forName", JClass, (JString, jboolean, JClassLoader), name, true, loader) end function mkclass(name::String, src::String) # based on https://stackoverflow.com/a/2946402 cls = mktempdir(; prefix="jj-") do root # write source to a file elems = split(name, ".") pkg_path = joinpath(root, elems[1:end-1]...) mkpath(pkg_path) src_path = joinpath(pkg_path, elems[end] * ".java") open(src_path, "w") do f write(f, src) end # compile jcompiler = jcall(JToolProvider, "getSystemJavaCompiler", JJavaCompiler) jcall(jcompiler, "run", jint, (JInputStream, JOutputStream, JOutputStream, Vector{JString}), nothing, nothing, nothing, [src_path]) # load class jfile = JFile((JString,), root) juri = jcall(jfile, "toURI", JURI) jurl = jcall(juri, "toURL", JURL) jloader = jcall(JURLClassLoader, "newInstance", JURLClassLoader, (Vector{JURL},), [jurl]) classforname(name, jloader) end return cls end function instantiate(name::String, src::String) jclass = mkclass(name, src) return jcall(jclass, "newInstance", JObject, ()) end JavaCall.init() name = "julia.compiled.Hello" src = """ package julia.compiled; public class Hello { public String hello(String name) { return "Hello, " + name; } public void goodbye() { System.out.println("Goodbye!"); } } """ obj = instantiate(name, src) jcall(obj, "hello", JString, (JString,), "Bob") # NoSuchMethodError: jcall(obj, "goodbye", Nothing, ()) # NoSuchMethodError: jcall(obj, "equals", jboolean, (JObject,), obj) # ok ```

[mkitti]

Well my point was the opposite actually. We need to use InMemoryJavaCompiler's dynamic classloader to load your classes, but JavaCall doesn't know anything about that. It just uses JNI's FindClass: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#FindClass

[dfdx]

But isn't FindClass only needed to locate and load the class? I thought after loading a class is self-containing, including all information required by JNI to work properly.

Would it help if we introduce a proxy class loaded using the system class loader (and thus JNI-friendly), but in runtime loading the new generated class?

[mkitti]

This is failing at https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetMethodID.

The question then is what are the correct arguments to pass to JavaCall.JNI.GetMethodID to get this to work? Or perhaps GetMethodID is not the right way to go.

Since reflection seems to be working, perhaps you should work off this result:

julia> listmethods(obj)
11-element Vector{JMethod}:
 java.lang.String hello(java.lang.String)
 void goodbye()
 void wait(long)
 void wait(long, int)
 void wait()
 boolean equals(java.lang.Object)
 java.lang.String toString()
 int hashCode()
 java.lang.Class getClass()
 void notify()
 void notifyAll()

v0.8.0rc-1 offers this new way of using jcall: https://github.com/JuliaInterop/JavaCall.jl/blob/63026e4cc6530c65b791519f34812e84ec843b14/src/core.jl#L374-L380

where get_method_id is implemented via JNI.FromReflectedMethod

https://github.com/JuliaInterop/JavaCall.jl/blob/63026e4cc6530c65b791519f34812e84ec843b14/src/core.jl#L396

Combined with the two argument form of listmethods you might be able to achieve what you want.

https://github.com/JuliaInterop/JavaCall.jl/blob/63026e4cc6530c65b791519f34812e84ec843b14/src/reflect.jl#L111-L114

[dfdx]

Just wanted to post an updated on this. I tried the suggested method, but it turns out due to @checknull any call to get_method_id on a non-existing method throws an exception and prints unwanted error message to console. This makes it impossible to create a robust side-effect-free version of jcall without rewriting half of its machinery. If I have a minute, I'll make a PR here, but it's not the top priority for me at the moment.

Also, I test it on OpenJDK 11, which I already had problems with in the past. Oracle download site for their JDK 11 is broken right now, once it's fixed, I'll check if this problem exists there too.

[mkitti]

Can you try to use JavaCall.JNI.* directly as if you were a C programmer? If you can demonstrate a string of JNI calls that work, then we can design a version of jcall that may work.

[mkitti]

Concretely, is there any other JNI method other than FromReflectedMethod that works? Does FromReflectedMethod work in all cases?

[dfdx]

I tried JNI.GetMethodID, JNI.GetStaticMethodID and even JNI.GetFieldID and JNI.GetStaticFieldID, but none of them worked. JNI.FromReflectedMethod works fine so far. The code I used:

Generate object (copy from above)

```julia # copy from one of the previous comments using JavaCall const JFile = @jimport java.io.File const JToolProvider = @jimport javax.tools.ToolProvider const JJavaCompiler = @jimport javax.tools.JavaCompiler const JInputStream = @jimport java.io.InputStream const JOutputStream = @jimport java.io.OutputStream const JClassLoader = @jimport java.lang.ClassLoader const JURLClassLoader = @jimport java.net.URLClassLoader const JURI = @jimport java.net.URI const JURL = @jimport java.net.URL function JavaCall.classforname(name::String, loader) return jcall(JClass, "forName", JClass, (JString, jboolean, JClassLoader), name, true, loader) end function mkclass(name::String, src::String) # based on https://stackoverflow.com/a/2946402 cls = mktempdir(; prefix="jj-") do root # write source to a file elems = split(name, ".") pkg_path = joinpath(root, elems[1:end-1]...) mkpath(pkg_path) src_path = joinpath(pkg_path, elems[end] * ".java") open(src_path, "w") do f write(f, src) end # compile jcompiler = jcall(JToolProvider, "getSystemJavaCompiler", JJavaCompiler) jcall(jcompiler, "run", jint, (JInputStream, JOutputStream, JOutputStream, Vector{JString}), nothing, nothing, nothing, [src_path]) # load class jfile = JFile((JString,), root) juri = jcall(jfile, "toURI", JURI) jurl = jcall(juri, "toURL", JURL) jloader = jcall(JURLClassLoader, "newInstance", JURLClassLoader, (Vector{JURL},), [jurl]) classforname(name, jloader) end return cls end function instantiate(name::String, src::String) jclass = mkclass(name, src) return jcall(jclass, "newInstance", JObject, ()) end JavaCall.init() name = "julia.compiled.Hello" src = """ package julia.compiled; public class Hello { public String hello(String name) { return "Hello, " + name; } public void goodbye() { System.out.println("Goodbye!"); } } """ obj = instantiate(name, src) ```

import JavaCall.JNI

sig = JavaCall.method_signature(JString, JString)
ptr = Ptr(JavaCall.metaclass(obj))

# methods
JNI.GetMethodID(ptr, "call", sig)           # ==> Ptr{Nothing} @0x0000000000000000
JNI.GetStaticMethodID(ptr, "call", sig)   # ==> Ptr{Nothing} @0x0000000000000000

# fields, just in case, with method signature
JNI.GetFieldID(ptr, "call", sig)               # ==> Ptr{Nothing} @0x0000000000000000
JNI.GetStaticFieldID(ptr, "call", sig)      # ==> Ptr{Nothing} @0x0000000000000000

# reflected method
meth = listmethods(obj, "hello")[1]
JNI.FromReflectedMethod(meth)         # ==> Ptr{Nothing} @0x00000000029d26a0