Macuyiko / minecraft-python

A Jython driven plugin and interpreter system for Minecraft (on top of Spigot)
BSD 3-Clause "New" or "Revised" License
227 stars 29 forks source link

Importing and using Java Libraries in lib-custom results in NoClassDefFoundError in 1.18 #58

Closed MuffinMario closed 2 years ago

MuffinMario commented 2 years ago

Hello,

after rigorously trying to fix the issue and comparing self-built 1.17 and 1.18 server and plugin jars, it seems to strongly indicate that importing jar libraries into Jython does not seem to work. I have tried multiple libaries; ones, that can not possible have been broken by the upgrade to 1.18, but it always results in the same error.

E.g. putting the PlayerBorderAPI into /plugins and /lib-custom, and running the following code

from com.github.zandy.playerborderapi.api import PlayerBorderAPI

results in this being shown:

[20:57:47] [Server thread/WARN]: Traceback (most recent call last):
[20:57:47] [Server thread/WARN]:   File "C:\Users\Administrator\Desktop\Work In Progress Project\paper 1.18.2\.\python-plugins\pluginloader.py", line 154, in <module>
[20:57:47] [Server thread/WARN]:     loadPlugins()
[20:57:47] [Server thread/WARN]:   File "C:\Users\Administrator\Desktop\Work In Progress Project\paper 1.18.2\.\python-plugins\pluginloader.py", line 115, in loadPlugins
[20:57:47] [Server thread/WARN]:     loadPlugin(filename[:-3], [], loaded_plugins)
[20:57:47] [Server thread/WARN]:   File "C:\Users\Administrator\Desktop\Work In Progress Project\paper 1.18.2\.\python-plugins\pluginloader.py", line 115, in loadPlugins
[20:57:47] [Server thread/WARN]:     loadPlugin(filename[:-3], [], loaded_plugins)
[20:57:47] [Server thread/WARN]:   File "C:\Users\Administrator\Desktop\Work In Progress Project\paper 1.18.2\.\python-plugins\pluginloader.py", line 79, in loadPlugin
[20:57:47] [Server thread/WARN]:     execfile(pluginname + ".py", globals())
[20:57:47] [Server thread/WARN]:   File "socialscoreboard.py", line 224, in <module>
[20:57:47] [Server thread/WARN]:     from com.github.zandy.playerborderapi.api import PlayerBorderAPI
[20:57:47] [Server thread/WARN]: java.lang.NoClassDefFoundError: org/bukkit/plugin/Plugin
[20:57:47] [Server thread/WARN]:        at java.base/java.lang.Class.forName0(Native Method)
[20:57:47] [Server thread/WARN]:        at java.base/java.lang.Class.forName(Class.java:467)
[20:57:47] [Server thread/WARN]:        at org.python.core.Py.loadAndInitClass(Py.java:1160)
[20:57:47] [Server thread/WARN]:        at org.python.core.Py.findClassInternal(Py.java:1095)
[20:57:47] [Server thread/WARN]:        at org.python.core.Py.findClassEx(Py.java:1147)
[20:57:47] [Server thread/WARN]:        at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:233)
[20:57:47] [Server thread/WARN]:        at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:36)
[20:57:47] [Server thread/WARN]:        at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:221)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:137)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyObject.__findattr__(PyObject.java:902)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyObject.__findattr__(PyObject.java:889)
[20:57:47] [Server thread/WARN]:        at org.python.core.imp.ensureFromList(imp.java:1484)
[20:57:47] [Server thread/WARN]:        at org.python.core.imp.ensureFromList(imp.java:1449)
[20:57:47] [Server thread/WARN]:        at org.python.core.imp.import_module_level(imp.java:1377)
[20:57:47] [Server thread/WARN]:        at org.python.core.imp.importName(imp.java:1528)
[20:57:47] [Server thread/WARN]:        at org.python.core.ImportFunction.__call__(__builtin__.java:1285)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyObject.__call__(PyObject.java:433)
[20:57:47] [Server thread/WARN]:        at org.python.core.__builtin__.__import__(__builtin__.java:1232)
[20:57:47] [Server thread/WARN]:        at org.python.core.imp.importFromAs(imp.java:1620)
[20:57:47] [Server thread/WARN]:        at org.python.core.imp.importFrom(imp.java:1595)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx4.f$0(socialscoreboard.py:271)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx4.call_function(socialscoreboard.py)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyTableCode.call(PyTableCode.java:173)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyCode.call(PyCode.java:18)
[20:57:47] [Server thread/WARN]:        at org.python.core.Py.runCode(Py.java:1687)
[20:57:47] [Server thread/WARN]:        at org.python.core.__builtin__.execfile_flags(__builtin__.java:535)
[20:57:47] [Server thread/WARN]:        at org.python.core.__builtin__.execfile(__builtin__.java:512)
[20:57:47] [Server thread/WARN]:        at org.python.core.__builtin__.execfile(__builtin__.java:539)
[20:57:47] [Server thread/WARN]:        at org.python.core.BuiltinFunctions.__call__(__builtin__.java:144)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyObject.__call__(PyObject.java:481)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx0.loadPlugin$7(C:/Users/Administrator/Desktop/Work In Progress Project/paper 1.18.2/./python-plugins/pluginloader.py:93)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx0.call_function(C:/Users/Administrator/Desktop/Work In Progress Project/paper 1.18.2/./python-plugins/pluginloader.py)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyTableCode.call(PyTableCode.java:173)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyBaseCode.call(PyBaseCode.java:168)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyFunction.__call__(PyFunction.java:437)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx0.loadPlugins$10(C:/Users/Administrator/Desktop/Work In Progress Project/paper 1.18.2/./python-plugins/pluginloader.py:140)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx0.call_function(C:/Users/Administrator/Desktop/Work In Progress Project/paper 1.18.2/./python-plugins/pluginloader.py)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyTableCode.call(PyTableCode.java:173)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyBaseCode.call(PyBaseCode.java:119)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyFunction.__call__(PyFunction.java:406)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx0.f$0(C:/Users/Administrator/Desktop/Work In Progress Project/paper 1.18.2/./python-plugins/pluginloader.py:156)
[20:57:47] [Server thread/WARN]:        at org.python.pycode._pyx0.call_function(C:/Users/Administrator/Desktop/Work In Progress Project/paper 1.18.2/./python-plugins/pluginloader.py)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyTableCode.call(PyTableCode.java:173)
[20:57:47] [Server thread/WARN]:        at org.python.core.PyCode.call(PyCode.java:18)
[20:57:47] [Server thread/WARN]:        at org.python.core.Py.runCode(Py.java:1687)
[20:57:47] [Server thread/WARN]:        at org.python.core.__builtin__.execfile_flags(__builtin__.java:535)
[20:57:47] [Server thread/WARN]:        at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:287)
[20:57:47] [Server thread/WARN]:        at com.macuyiko.minecraftpyserver.jython.JyInterpreter.execfile(JyInterpreter.java:128)
[20:57:47] [Server thread/WARN]:        at com.macuyiko.minecraftpyserver.MinecraftPyServerPlugin.startPluginInterpreters(MinecraftPyServerPlugin.java:123)
[20:57:47] [Server thread/WARN]:        at com.macuyiko.minecraftpyserver.MinecraftPyServerPlugin.onEnable(MinecraftPyServerPlugin.java:58)
[20:57:47] [Server thread/WARN]:        at org.bukkit.plugin.java.JavaPlugin.setEnabled(JavaPlugin.java:264)
[20:57:47] [Server thread/WARN]:        at org.bukkit.plugin.java.JavaPluginLoader.enablePlugin(JavaPluginLoader.java:342)
[20:57:47] [Server thread/WARN]:        at org.bukkit.plugin.SimplePluginManager.enablePlugin(SimplePluginManager.java:480)
[20:57:47] [Server thread/WARN]:        at org.bukkit.craftbukkit.v1_18_R2.CraftServer.enablePlugin(CraftServer.java:518)
[20:57:47] [Server thread/WARN]:        at org.bukkit.craftbukkit.v1_18_R2.CraftServer.enablePlugins(CraftServer.java:432)
[20:57:47] [Server thread/WARN]:        at net.minecraft.server.MinecraftServer.loadWorld0(MinecraftServer.java:612)
[20:57:47] [Server thread/WARN]:        at net.minecraft.server.MinecraftServer.loadLevel(MinecraftServer.java:414)
[20:57:47] [Server thread/WARN]:        at net.minecraft.server.dedicated.DedicatedServer.e(DedicatedServer.java:263)
[20:57:47] [Server thread/WARN]:        at net.minecraft.server.MinecraftServer.w(MinecraftServer.java:1007)
[20:57:47] [Server thread/WARN]:        at net.minecraft.server.MinecraftServer.lambda$0(MinecraftServer.java:304)
[20:57:47] [Server thread/WARN]:        at java.base/java.lang.Thread.run(Thread.java:833)
[20:57:47] [Server thread/WARN]: Caused by: java.lang.ClassNotFoundException: org.bukkit.plugin.Plugin
[20:57:47] [Server thread/WARN]:        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
[20:57:47] [Server thread/WARN]:        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
[20:57:47] [Server thread/WARN]:        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
[20:57:47] [Server thread/WARN]:        ... 61 more
[20:57:47] [Server thread/WARN]: java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/bukkit/plugin/Plugin

From this post it seems, that in Java 17 the way Reflections work has changed. Though I have not looked into how this plugin functions, but maybe that could be a hint?

Macuyiko commented 2 years ago

Should be fixed in the latest release. Reflections and JDK 17+ was already fixed previously. The problem was due to the way how Spigot now handles its own JARs/libraries.

MuffinMario commented 2 years ago

It has definitely improved the situation. There is still an error if you pass interface implementations through custom libraries, it will tell that the passed object cannot be coerced to the interface.

E.g. ScoreboardLib cannot work, until you set the plugin you are implementing the API in with ScoreboardLib.setPluginInstance(org.bukkit.plugin.Plugin). Originally this worked with 1.17 by using

from mcapi import *
from me.tigerhix.lib.scoreboard import ScoreboardLib

def onEnable():
    ScoreboardLib.setPluginInstance(PLUGIN) 

but now, it will only tell that the object PLUGIN cannot be coerced to 'org.bukkit.plugin.Plugin' when running the last line, albeit type(PLUGIN) returning the Object of this plugin, which is MinecraftPyServerPlugin, which is still extending JavaPlugin, which is still extending PluginBase, which is still implementing said interface Plugin (org.bukkit.plugin.Plugin).

edit: while, yes, in this example, you could just let the plugin run as its own instance, it will still proceed to not be able to coerce any bukkit/(spigot/paper?) type from any of these types that you pass it to

Macuyiko commented 2 years ago

Thanks for reporting this. The new release should hopefully make some strides towards improving this.

MuffinMario commented 2 years ago

The issues mentioned above don't appear anymore; although I cannot fully test all my functionalities. The new implementation seems to mess with how Java originally reflects the classes in its memory (?). Initializing a class' static data (via calling Class.forName(name) which name is outside of the java standard library will result in a ClassNotFoundException. Importing the class and calling new Obj() via dummy = Obj() "works" as in it will run and find the class, but does not solve the issue either.

See

[00:38:46 WARN]:     Class.forName("org.sqlite.JDBC")
[00:38:46 WARN]:        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
[00:38:46 WARN]:        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
[00:38:46 WARN]:        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
[00:38:46 WARN]:        at java.base/java.lang.Class.forName0(Native Method)
[00:38:46 WARN]:        at java.base/java.lang.Class.forName(Class.java:375)
[00:38:46 WARN]:        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[00:38:46 WARN]:        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[00:38:46 WARN]:        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[00:38:46 WARN]:        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[00:38:46 WARN]: java.lang.ClassNotFoundException: java.lang.ClassNotFoundException: org.sqlite.JDBC

I've tried and found these results:

Class.forName("org.sqlite.JDBC") # ClassNotFoundException
Class.forName("me.tigerhix.lib.scoreboard.ScoreboardLib") # ClassNotFoundException
Class.forName("org.bukkit.Location") # ClassNotFoundException
Class.forName("org.bukkit.ChatColor") # ClassNotFoundException
Class.forName("java.util.UUID") # works
Class.forName("java.lang.Class") # works
Macuyiko commented 2 years ago

Edit:

from org.bukkit import Bukkit
from java.lang import Class
Class.forName("org.bukkit.Location", True, Bukkit.getClassLoader())

Works. Most likely there is a class loader conflict in Jython.

MuffinMario commented 2 years ago

With Bukkit.getClassLoader().loadClass or the three-parameter Class.forName, it finds the class and loads it. Presumably in a space, that the default java library cannot find. Using either of those options will result in

[01:33:26 WARN]:        at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:706)
[01:33:26 WARN]:        at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:252)
[01:33:26 WARN]:        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[01:33:26 WARN]:        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[01:33:26 WARN]:        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[01:33:26 WARN]:        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[01:33:26 WARN]: java.sql.SQLException: java.sql.SQLException: No suitable driver found for jdbc:sqlite:testdb.db

which for sqlite was the same exception occuring before.

Maybe the class data loaded by the class loader from bukkit is being loaded into an environment that differs from the environment where the default java api is located in?

Macuyiko commented 2 years ago

With the latest release, this should now work:

# Making sure mcapi works and libs in lib-custom:
from mcapi import *
from com.github.zandy.playerborderapi.api import PlayerBorderAPI
from me.tigerhix.lib.scoreboard import ScoreboardLib
ScoreboardLib.setPluginInstance(PLUGIN)

# Importing classes
import sys
from java.lang import Class
from org.python.core import Py

Class.forName("org.sqlite.JDBC") # Works => uses the system classloader
Class.forName("org.bukkit.Location") # Doesn't work => system classloader not aware of this class
Class.forName("org.bukkit.Location", True, sys.getClassLoader()) # Works => jython classloader
Py.findClass("org.bukkit.Location") # Works => jython classloader (and shorter to write)

# Connection
from java.sql import DriverManager
conn = DriverManager.getConnection("jdbc:sqlite::memory:") # Works => since we loaded org.sqlite.JDBC using the system classloader
MuffinMario commented 2 years ago

For now, it seems that everything is working. Thank you for your quick responses and cooperation!