Closed jonathanindig closed 1 year ago
That is bad. You have accurately identified the change that causes the problem. The goal of that change was to create a Python type hierarchy that mirrors the Java class hierarchy. The change was mostly implemented as a way to fix #307 by allowing a PyJObject to inherit special python functionality from multiple Java interfaces. Conceptually there are other features I was hoping to build off the mirrored type hierarchy.
I am not very familiar with how scala classes work and specifically how they are translated into Java classes but I have managed to get the same error using only Java interfaces, as shown below. I suspect if I can resolve this use case then the Scala use case will also be resolved.
class Test {
public static interface Root {
}
public static interface Child extends Root{
}
public static class ProblemClass implements Root, Child {
}
public static void main(String[] args) throws JepException {
try (SharedInterpreter interpreter = new SharedInterpreter()) {
interpreter.set("test", new ProblemClass());
}
}
}
The problem is that I assumed that any type hierarchy in Java would also be a valid Python type hierarchy. I can easily demonstrate that this is not the case just by reproducing the problematic Java hierarchy in pure Python, shown below. Whether the type hierarchy is defined in Java or Python it violates the default Python Method Resolution order(MRO) and cannot be represented in Python. Fortunately it is possible to define a custom MRO by defining a metaclass. I will need to do more research but I think it would be acceptable to use a custom metaclass for all the pyjtypes which would use a simplified MRO. I do not think the actual MRO used by Python is important to the functionality of pyjtype since each type currently includes all methods defined by supertypes(which is necessary to handle overloaded methods) and JNI is actually handling the method resolution.
>>> class Root: pass
>>> class Child(Root): pass
>>> class ProblemClass(Root,Child): pass
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Cannot create a consistent method resolution
order (MRO) for bases Root, Child
I've made an initial attempt at a custom MRO that works for my small test case and submitted it as #407, The scala class hierarchy is much more interesting than my small example so it would be fantastic if you can test this change with scala to see if I have correctly identified the problem.
Amazing @bsteffensmeier , thanks so much for your quick response! I will try to test this out today.
@bsteffensmeier I set up a test in Scala with some crazy type hierarchies (including testing List
, which has a pretty complex hierarchy).
This test throws exceptions with Jep 4.0.3 (Note the a[JepException] shouldBe thrownBy
assertions).
It looks like your changes in #407 fix the problem though, as I no longer get exceptions after installing your branch with pip3 install git+https://github.com/bsteffensmeier/jep.git@pyjtype_mro
!
@jeremyrsmith can you think of any other weird inheritance hierarchies that Scala supports that aren't covered by the test I wrote?
Thanks for taking the time to test it, I am glad I solved the right problem instead of finding a new one.
I am not sure if there is a better MRO algorithm to use for Java classes than the one I proposed. Since Java mostly ignores interfaces I don't think the MRO really matters as long as the non-interface classes are in order but I'd like to give it more thought before merging, If anyone else has thoughts I am open to ideas, especially if there any other cases where scala might show problems more easily than java alone.
@jonathanindig the only weird Scala-specific thing I can think of is abstract override
. But, in a concrete class, that will end up being encoded in ways that are pretty much tested by your code.
IIRC, in Scala 2.13 (and maybe 2.12?), there is some further Java-8-ification of certain trait methods as interface default methods. So it might be worth adding some default method cases to the test in #407 @bsteffensmeier (I'll comment there).
Thanks for looking into this so quickly!
This was fixed in jep 4.1
We're running into an exception with Jep 4.0.0+ in Polynote when certain Scala types are made available to Jep. For example, here's what we get with a Scala
List
type:I have a hunch that this may be related to https://github.com/ninia/jep/commit/151c748f6c3674bc1ef292156fa82224a208fa3c, because I see that there's some Java class inheritance inspection stuff going on.
The same code works with Jep 3.9.1, so I know it's something new in 4.0.0. I tried with Python 3.7 and 3.9.
Hopefully this is enough to go by 😬 If not, I can try to come up with a minimal Scala repro.
FYI @jeremyrsmith