Adding JProxy, a wrapper that allows Java-like syntax for field and method access

[zot]

This lets you say things like:

julia> a=JProxy(@jimport(java.util.ArrayList)(()))
[]

julia> a.size()
0

julia> a.add("hello")
true

julia> a.get(0)
"hello"

julia> a.isEmpty()
false

julia> a.toString()
"[hello]"

julia> b = a.clone()
[hello]

julia> b.add("derp")
true

julia> a == b
false

julia> b == b
true

julia> JProxy(@jimport(java.lang.System)).getName()
"java.lang.System"

julia> JProxy(@jimport(java.lang.System);static=true).out.println("hello")
hello

Note that a.clone() returns a JProxy, so you can use fields and methods on objects from fields and methods.

[aviks]

Thanks Bill, this is amazing.

I want to spend some time thinking about the interface. is JProxy(@jimport(..... the right abstraction. Or rather, is @jimport the best way to begin with. A few years ago it seemed like the most efficient interface, but I want to take this opportunity to re-think that.

But either ways, this kind of functionality is something I've wanted for a long time. Thanks for this, its very useful.

[zot]

Thanks!

It would be possible to make JProxy the main API, yes, but I didn't do that because I didn't want to make that level of decision for your project :). It would be possible to merge JProxy's functionality into JavaObject and remove the JProxy type.

JProxy currently uses an "interpretive" approach, which is good enough for a first cut, I think.

A next step might be to define a function for each method when a class is first discovered. JMethod proxy could become a value type parameterized on the method name and defining class, so the Julia function for a method could be something like this:

function (pxy::JMethodProxy{Symbol("java.util.ArrayList"), :remove})(i::int32)
    ...
end

This would remove the need for argument type conversion and also let Julia do the method dispatch instead of using reduce with a "generality" function, like I'm doing right now. It would also remove decision making for the result since the type is known at compile time.

To fit into the Julia model more idiomatically, it could also define an explicitly named method. Then both remove(arrayList, 1) or arrayList.remove(1) would work.

[zot]

Thinking about it, I think the class parameter I proposed for JMethodProxy can't be just a symbol, it would have to be a type itself and the system would have to generate a type hierarchy parallel to Java's class hierarchy.

I'm going to start working on the compiled version of JProxy in another branch.

[aviks]

It would be possible to merge JProxy's functionality into JavaObject

I would be up for that.

I'm going to start working on the compiled version of JProxy in another branch.

Thanks for running with this.

paging @dfdx and @ExpandingMan for their thoughts?

[zot]

Sure thing, grafting things onto other things is my bag, man.

Right now I'm focusing on code cleanup and using compilation instead of dynamic techniques for methods.

After that I'll look at merging functionality into JavaObject and removing the JProxy type.

[ExpandingMan]

Awesome, thanks so much for this! It's amusing that we'll have this ability here before PyCall as that package is much more widely used.

I'm all for making getproperty the main interface for all JavaObjects, I can't see any advantage of having JProxy as a separate object (though thank you for the consideration that went into making that choice). That would of course be a big change and we'd have to be really careful as many tests would probably have to be rewritten.

I'm definitely not a Java expert, I got involved in this package mostly because I needed JDBC.jl to be working reliably, so I certainly don't have any useful insight to how this will all wind up working. I suppose the way to go would be to keep jcall as a "low-level" interface but have it so that most of the time users only need @jimport and getproperty.

Anyway, thanks again! I'm sure having this will be a huge relief for those working on packages like JDBC.jl and Spark.jl.

[dfdx]

A next step might be to define a function for each method when a class is first discovered.

We should be very careful not to run into method generation problem in Julia. Say, if we have:

function foo()
    # first import of JSomeObject, defines its methods
    jobj = JProxy(@jimport JSomeObject)
    jobj.bar()
end

bar() may be not defined in the generation that foo() is running in. I'm also not sure there will be no method definition conflicts.

To fit into the Julia model more idiomatically, it could also define an explicitly named method. Then both remove(arrayList, 1) or arrayList.remove(1) would work.

It may be quite troublesome if you already have remove in scope. Say, you have:

using A     # export function `remove()`

# implicitly generated from @jimport
function remove(arr::JArrayList, i::Int32)
    ...
end

Later (implicit) definition would hide exported one, leading to hard-to-debug issues.

Some possible fixes:

1. Define something like j_remove(arr::JArrayList, i::Int32) to decrease probability of name collision. Note, however, that Java and Julia have different naming convention, so neither doSomeStuff, nor j_doSomeStuff follow Julia conventions which would name it do_some_stuff.
2. Don't mess with method definition and just keep cache of (static_flag, method_name, arg_types) => method_instance pairs inside of jimported class object.

I also hope that we retain old API based on jcall as well - not only a lot of code have already been written using it, but also its much easier to debug: while working on Spark.jl I found enough JNI weirdness to appreciate ability to work with the most low-level API I can get.

[zot]

This is a good point, mirroring parts of an entire language and SDK will almost guarantee naming conflicts . I'll just generate proxy-style functions only and not named ones. That will guarantee no naming conflicts.

function (pxy::JMethodProxy{Symbol("wait"), <:java_lang_Object})(a1::Int64)
        _jcall(getfield(pxy, :obj), (methodsById[1]).id, C_NULL, Int32, (Int64,), a1)
    end

A next step might be to define a function for each method when a class is first discovered.

We should be very careful not to run into method generation problem in Julia. Say, if we have:

function foo()
    # first import of JSomeObject, defines its methods
    jobj = JProxy(@jimport JSomeObject)
    jobj.bar()
end

bar() may be not defined in the generation that foo() is running in. I'm also not sure there will be no method definition conflicts.

To fit into the Julia model more idiomatically, it could also define an explicitly named method. Then both remove(arrayList, 1) or arrayList.remove(1) would work.

It may be quite troublesome if you already have remove in scope. Say, you have:

using A     # export function `remove()`

# implicitly generated from @jimport
function remove(arr::JArrayList, i::Int32)
    ...
end

Later (implicit) definition would hide exported one, leading to hard-to-debug issues.

Some possible fixes:

1. Define something like j_remove(arr::JArrayList, i::Int32) to decrease probability of name collision. Note, however, that Java and Julia have different naming convention, so neither doSomeStuff, nor j_doSomeStuff follow Julia conventions which would name it do_some_stuff.
2. Don't mess with method definition and just keep cache of (static_flag, method_name, arg_types) => method_instance pairs inside of jimported class object.

I also hope that we retain old API based on jcall as well - not only a lot of code have already been written using it, but also its much easier to debug: while working on Spark.jl I found enough JNI weirdness to appreciate ability to work with the most low-level API I can get.

[zot]

I have the compiled version of the proxy mostly working here. I still have to do array conversion. Here are some example generated methods:

    function (pxy::JMethodProxy{Symbol("contains"), <:java_util_AbstractCollection})(a1::Union{Number, String, JProxy})
        _jcall(getfield(pxy, :obj), (methodsById[30]).id, C_NULL, UInt8, (JavaObject{Symbol("java.lang.Object")},), box(a1)) != 0
    end
    function (pxy::JMethodProxy{Symbol("add"), <:java_util_AbstractList})(a1::Number, a2::Union{Number, String, JProxy})
        _jcall(getfield(pxy, :obj), (methodsById[44]).id, C_NULL, Int32, (Int32, JavaObject{Symbol("java.lang.Object")}), (Int32)(a1), box(a2))
    end
    function (pxy::JMethodProxy{Symbol("get"), <:java_util_ArrayList})(a1::Number)
        asJulia(JavaObject{Symbol("java.lang.Object")}, _jcall(getfield(pxy, :obj), (methodsById[67]).id, C_NULL, Any, (Int32,), (Int32)(a1)))
    end
    function (pxy::JMethodProxy{Symbol("get"), <:java_util_AbstractList})(a1::Number)
        asJulia(JavaObject{Symbol("java.lang.Object")}, _jcall(getfield(pxy, :obj), (methodsById[47]).id, C_NULL, Any, (Int32,), (Int32)(a1)))
    end

I'm generating a parallel type hierarchy in the JavaCall module with names like java_util_AbstractList. As you can see, I'm still using _jcall but _jcall makes some decisions which are known at compile time so I think that can be made more efficient or at least it could call a specialized version of _jcall.

A few things:

• asJulia() is only used if needed
• booleans are converted in-line (as in ArrayList.contains())
• numeric arguments are presented as Number and converted in the body
• Object arguments are presented as Union{Number, String, JProxy} and converted in the body

[zot]

I'm still working on this -- done a few overhauls. Lately I realized that JavaObjects use LocalRefs and JProxy really needs to use GlobalRefs and it looks like that means JProxy should use Ptr{Nothing} instead of JavaObject.

So JProxy would be an alternative interface instead of a wrapper but there would still be a simple way to convert between JProxy and JavaObject, like JProxy(jobj) and JavaObject(pxy)

[zot]

Sorry this is taking so long -- dealing with a ton of RL interrupts. I just committed a bunch of changes to my branch in my repo: https://github.com/zot/JavaCall.jl/tree/jproxy-compiled I'll keep you guys informed about progress.

[zot]

• I reworked the design so that JProxy now has a Ptr{Nothing} instead of a JavaObject. Dynamic proxy calls and field gets/sets work but it still doesn't handle array conversion yet.

• I think implicitly converting Julia array arguments is good but I think developers will want to choose whether to convert returned arrays -- you wouldn't want to convert returned arrays that are simply passed back to Java as arguments and not used in Julia code.

• JavaObject now uses global references and automatically deallocates any local references

• Code generation is optional and should be a much more efficient alternative to the dynamic proxy behavior but I still need to convert code generation to use the new design though, so it doesn't work yet. gen(x) generates code, where x can be a proxy, class, symbol, or string and specifies the class for which to generate code (if the code has not already been generated).

[dfdx]

I think implicitly converting Julia array arguments is good but I think developers will want to choose whether to convert returned arrays -- you wouldn't want to convert returned arrays that are simply passed back to Java as arguments and not used in Julia code.

As far as I remember, we had this question for Julia/Java strings as well. For strings we decided to distinguish between 2 cases using return type: convert to Julia type if return type is String and keep pointer to Java object if it's JString. For arrays we couldn't do it since there's no explicit Java type for arrays (although we could introduce pseudo-type). I'm curious what's your vision about this conversion API?

[zot]

It would be possible to annotate the class information structure with advice about type conversion on a per-method basis. Right now the proxy just automatically acquires its methods from Java and handles everything the same way.

At this point it always converts strings to and from Julia, treating them like primitive types.

I changed the type information to progress some more towards automatic conversion for array parameters. I checked the code generation for my conversion testing and it's very tight:

julia> J.canConvert(J.java_lang_Integer, 3)
true

julia> @code_native J.canConvert(J.java_lang_Integer, 3)
    .text
; Function canConvert {
; Location: proxy.jl:1147
    movb    $1, %al
    retq
    nopw    %cs:(%rax,%rax)
;}

Note that this is using the generated type hierarchy, not JavaObject{T} because the type hierarchy mirrors Java's class hierarchy. This doesn't handle interfaces at this point but I'll be adding that soon.

[aviks]

I think developers will want to choose whether to convert returned arrays

The way pycall does it, I believe is that is the method to call is provided as a Symbol, the return value is converted, but if it is provided as a string, it is returned as a wrapped pointer.

[zot]

We could do something like that. Right now, you can say:

a = JProxy(@jimport(java.util.ArrayList)(()))
a.add("hello")
a.add(1)
a.get(0) == "hello"
a[1] == "hello"
a[2] == 1

a.add and a.get return method proxy structures, not functions, so we could support operations on them, maybe like call(a.get, 0, rawreturn=true) and call(a.add, ptr, rawargs=true).

[zot]

Still working on this. I have a memory leak now that I'm trying to track down.

[zot]

Made a bunch of progress -- seem to have the memory leak fixed. Problem with boxing now.

It FEELS like I'm close :)

[zot]

All tests work now (including the one boxing test). More tests would be good.

Also, the local reference management code I put in allowed me to remove the GC calls from the tests.

[zot]

OK, added a test that demonstrates that interface arguments work. Made two arraylists, a and b. Added 1 to a and then called b.addAll(a). Arraylist.addAll() takes a Collection argument, which is an interface. I hand-tested that it failed with b.addAll(1). I'll add an exception test for that.

So this clears the way for code generation. I moved my old generation code to another file and I'll use that as a starting point. It's FAIRLY complete except that it needs to use a dispatch method to deal with interface arguments.

YAY!

[PallHaraldsson]

[off-topic comment on]

Awesome, thanks so much for this! It's amusing that we'll have this ability here before PyCall as that package is much more widely used.

Actually PyCall has this ability (PR just isn't merged, only waiting for it):

https://github.com/JuliaPy/PyCall.jl/pull/517#issuecomment-439010331

That PR is older, since August, but at least by now, I see it confirmed you can use that branch; nothing to do to merge it, only rebase.

Still great to have this here. This or Rcall.jl it the next most important package to have this dot overloading for OO. Actually I haven't looked into syntax of OO in R (or in MATLAB), so I'm not sure if needed there or in MATLAB.jl. Are there other packages that need such?

[kcajf]

@zot really looking forward to using this - thanks for all the work you've put in.

One question - and forgive me if this is obvious - but why does this functionality need to be exposed via the JProxy wrapper? Is there a major reason why the dot syntax could not apply directly to the JObjects themselves? It seems like once this merges most use-cases would be easier with JProxy, so every constructor call will end up looking like JProxy(@jimport(java.util.ArrayList)(())).

[zot]

@zot really looking forward to using this - thanks for all the work you've put in.

No problem! My main motivation is that I spend most of my day working in Java and I want to use Julia as a beefed up developer console for our product :). I prefer Julia to Java so this is one way I can get to use it for work. Also, I like connecting things to other things.

One question - and forgive me if this is obvious - but why does this functionality need to be exposed via the JProxy wrapper? Is there a major reason why the dot syntax could not apply directly to the JObjects themselves? It seems like once this merges most use-cases would be easier with JProxy, so every constructor call will end up looking like JProxy(@jimport(java.util.ArrayList)(())).

Three reasons:

1. I didn't want to change the way JavaObject works without permission, if whoever owns this code wants me to merge them, I'm happy to do that.
2. Right now, JProxy uses JavaObject during initialization but I'm already starting to remove the dependencies, once I'm finished removing those, I could merge JProxy into JavaObject and remove JProxy.
3. I wanted to keep JavaObject stable during development so I had a baseline to check against. Now that JProxy seems to be mostly working, I don't need a baseline anymore :).

To create instances now, btw, you can use the @class() macro which creates a static proxy that can act like a function. So to make an array list you can say @class(java.util.ArrayList)() or you could make a variable const ArrayList = @class(java.util.ArrayList) and just say ArrayList()

You can also use them to access static members, like @class(java.lang.Integer).MAX_VALUE. You can use a proxy on a class object as a constructor, too, like anArrayList.getClass()() but since it's a proxy on a class object, you can't use it for static members although you can use it for getName() and getDeclaredMethods(), just like in Java.

Looking at the name though, I'm thinking now that @static is a much better name than @class.

[zot]

OK, tests work fine on one of my machines which has OpenJDK 1.8.0_191-8u191-b12-0ubuntu0.18.10.1-b12 but fail on the machine that has Oracle's JDK 1.8.0_144 in a way similar to the Travis failure. Testing with that now...

[zot]

Have had some RL interrupts here -- heading for a 2-week trip though during which I plan to try to get the tests working on both machines. I don't think I'll have much Internet access though...

[zot]

I updated to JDK 1.8.0_191 and the tests work on the machine that was failing earlier. It seems to be the same problem the Travis box is having.

[zot]

I got slammed with obligations and I don't see a lot of spare time in the near future. This is quite functional but I think it still needs a little more love. I'd greatly appreciate it if someone would be willing to jump in and help out here...

[dfdx]

I hoped to take a look at it on the weekend, but it seems like another project will take me busy for some time. Let's keep it open for now - this PR won't be lost anyway, so we can just wait for someone with related project to get to it (e.g. on my next iteration with Spark.jl).

[schlichtanders]

Just stumbled upon JavaCall.jl and looked for a bit more convenient way to call java than using jcall. Was hoping to find a string macro, like python R and cxx support it.

Then I found this and it looks quite nice, what is the status? Is there currently any alternative to jcall?

[zot]

I haven't worked on it for quite a while -- I was hoping someone might step up to help out with it...

[mkitti]

I have resolved conflicts on this PR at #112. Tests are passing there including test added for JProxy

[aviks]

Thanks @mkitti for picking this up. This is a lot of code, a pretty major re-write of of this package, and I was not sure of having the time and effort to maintain this without the help of the original author of these changes, even though I've long wanted this functionality. Hopefully with Mark's help we can take this to it's conclusion.

[zot]

Thanks @mkitti for picking this up. This is a lot of code, a pretty major re-write of of this package, and I was not sure of having the time and effort to maintain this without the help of the original author of these changes, even though I've long wanted this functionality. Hopefully with Mark's help we can take this to it's conclusion.

Yeah, I know it's a ton of stuff, sorry about that -- rather than taking an "interpretive approach" and just delegating arguments, I took a "compiled approach" and generated methods with typed parameters in an effort to allow better code generation and error detection. Also, I'm not a super experienced Julia programmer so I'm not positive my techniques are the best but I figure something is better than nothing :)

Although I don't have time at this point to actually be responsible for maintaining the code, I'm still more than happy to help!

[mkitti]

The direction we should take on this is to deploy this as a distinct package in a subdirectory of this repository that depends on core JavaCall.

The task now is to figure out a minimal patch that needs to be made to JavaCall itself and what can be put into an accessory package.

[zot]

The direction we should take on this is to deploy this as a distinct package in a subdirectory of this repository that depends on core JavaCall

That sounds like a good idea to me, I was never intending to replace JavaCall