Context classloader fails to load class from the shell.

[milleruntime]

Describe the bug Context classloader fails to load class from the shell.

Versions (OS, Maven, Java, and others, as appropriate):

• Affected version(s) of this project: 2.1

To Reproduce Follow the instructions for the classpath example: https://github.com/apache/accumulo-examples/blob/main/docs/classpath.md

Expected behavior The filter should be loaded by the shell for the iterator specified.

Additional context The shell reports: 2021-03-25T15:55:23,854 [shell.Shell] ERROR: org.apache.accumulo.shell.ShellCommandException: Command could not be initialized (Unable to load org.apache.accumulo.test.FooFilter; class not found.)

The tserver log is aware of the context and the jar in HDFS but doesn't report any errors:

2021-03-26T08:17:23,999 [vfs.ContextManager] DEBUG: ClassLoader not created for context cx1, creating new one. uris: hdfs://localhost:8020/user1/lib/FooFilter.jar, preDelegation: true
2021-03-26T08:17:24,002 [vfs.AccumuloVFSClassLoader] DEBUG: Resolving path element: hdfs://localhost:8020/user1/lib/FooFilter.jar
2021-03-26T08:17:24,063 [vfs.AccumuloReloadingVFSClassLoader] DEBUG: monitoring hdfs://localhost:8020/user1/lib/FooFilter.jar
2021-03-26T08:17:24,064 [vfs.ContextManager] DEBUG: Returning classloader  for context cx1
2021-03-26T08:18:05,043 [vfs.ContextManager] DEBUG: Managed Contexts: {cx1=org.apache.accumulo.start.classloader.vfs.ContextManager$Context@55348174}
2021-03-26T08:18:05,043 [vfs.ContextManager] DEBUG: Configured Contexts: [cx1]

The original bug in the examples repo is here: https://github.com/apache/accumulo-examples/issues/66

I believe the changes to deprecate the VFS classloader may have broke the expected behavior. This was done in: https://github.com/apache/accumulo/pull/1715

[ctubbsii]

Could try to add a debug statement in ClassLoaderUtil.getClassLoader(context) to verify the shell is correctly obtaining the context from the table config and it is reaching this code with the correct context, and to check the returned ClassLoader class name().

[milleruntime]

The obfuscation in the classloading code makes it difficult to debug what is going on. I was seeing a null context on the server side but I think that was a red herring as it appears it wasn't making it to the server code. I finally just printed the exception in the shell:

java.lang.ClassNotFoundException: org.apache.accumulo.test.FooFilter
    at java.net.URLClassLoader.findClass(URLClassLoader.java:471) ~[?:?]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:589) ~[?:?]
    at org.apache.accumulo.start.classloader.AccumuloClassLoader$1.loadClass(AccumuloClassLoader.java:212) ~[accumulo-start-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
    at org.apache.accumulo.shell.commands.SetIterCommand.setUpOptions(SetIterCommand.java:211) ~[accumulo-shell-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
    at org.apache.accumulo.shell.commands.SetIterCommand.execute(SetIterCommand.java:103) ~[accumulo-shell-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
    at org.apache.accumulo.shell.Shell.execCommand(Shell.java:779) ~[accumulo-shell-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
    at org.apache.accumulo.shell.Shell.start(Shell.java:598) ~[accumulo-shell-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
    at org.apache.accumulo.shell.Shell.execute(Shell.java:528) ~[accumulo-shell-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
    at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:126) ~[accumulo-start-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]

I am not sure why the URLClassLoader is being used in this case as I would assume DefaultContextClassLoaderFactory would be the default.

[ctubbsii]

The obfuscation in the classloading code makes it difficult to debug what is going on.

That complexity was one of the motivations for moving the classloading code out of Accumulo and supporting its features in a different way. The transition period is a bit rough, though, because we still have to support the existing behavior with the existing config. But, once we can drop these deprecated classes (presumably 3.0?), a lot of this code will make a lot more sense and these features will be much more maintainable.

I am not sure why the URLClassLoader is being used in this case as I would assume DefaultContextClassLoaderFactory would be the default.

In order to preserve the existing behavior, the DefaultContextClassLoaderFactory currently delegates to the old AccumuloVFSClassLoader, which in turn, uses AccumuloClassLoader as the parent of any VFS classloaders it creates. This parent AccumuloClassLoader is little more than a wrapper around URLClassLoader designed to handle the URLs in the (now deprecated) general.classpaths property. So, URLClassLoader makes sense here. What doesn't make sense is why it's using the standard classloader, rather than the context-specific classloader.

It's possible it's because the context for the table is being set after the table is created (which is consistent with you seeing null for the context), and the property is being read from an outdated ZooCache from the table's initial creation. Instead of setting the table property after the table is created, you could try setting it at creation time with createtable -prop table.classpath.context=cx1 nofoo

[dlmarion]

I recently tested the classloaders implementation in 2.1 using the procedure at https://github.com/apache/accumulo-classloaders/blob/main/modules/vfs-class-loader/TESTING.md

[milleruntime]

It's possible it's because the context for the table is being set after the table is created (which is consistent with you seeing null for the context), and the property is being read from an outdated ZooCache from the table's initial creation.

That would explain why I saw it work once after running another example, assuming I forced the ZooCache to update.

[jmark99]

I created a new instance and tried @ctubbsii suggestion but ended up with same result (class not found).

[milleruntime]

I think I found the issue. The problem was with one of the methods in the Shell that got the table context. See #1979