Concerning warnings about missing classes when using JanusGraph 1.0.0 with CQL in embedded mode

[pete-gillin-privitar]

Discussed in https://github.com/JanusGraph/janusgraph/discussions/4349

^{Originally posted by **pete-gillin-privitar** March 27, 2024} We are using JanusGraph in embedded mode with the CQL backend. On upgrading to JanusGraph 1.0.0, we are getting some concerning warnings about missing classes when we do our first traversal. I am not sure whether I should be worried about these, or what combination of maven dependencies (and possibly exclusions?) I need to get rid of them. I can reproduce this by doing anything which causes the class `JanusGraphTraverserUtil` to be loaded, in a maven project with `org.janusgraph:janusgraph-core` and `org.janusgraph:janusgraph-cql` as dependencies. I am providing the full text of the warnings and a minimal standalone reproduction below, but I'll summarize quickly here. We have a dependency on `janusgraph-cql` but not on `janusgraph-hadoop`. That means that we have `org.janusgraph.hadoop.scan.CQLHadoopScanRunner` on the classpath, but not `org.janusgraph.hadoop.scan.AbstractHadoopScanRunner` which `CQLHadoopScanRunner` extends. That means that we'll never be able to load the class `CQLHadoopScanRunner`. That doesn't seem to matter in itself, since (in our testing so far) we have not hit a code path which actually uses that class. _However_, the first time we do a traversal, the static initializer of `org.janusgraph.graphdb.util.JanusGraphTraverserUtil` uses the `Reflections` utility to scan every class under `org.janusgraph`, and that tries to expand the supertypes, and that logs a warning `could not get type for name org.janusgraph.hadoop.scan.AbstractHadoopScanRunner from any class loader`. We get a similar story for a few other classes (see the below). As well as several from `janusgraph-hadoop`, this includes one in `org.apache.hadoop:hadoop-mapreduce-client-core`, and one in `org.apache.tinkerpop:gremlin-server`. The [upgrade instructions](https://docs.janusgraph.org/master/changelog/#removal-of-cassandra-all-dependency) contain this note, which seems potentially relevant: > If you are running embedded JanusGraph with Cassandra, you have to exclude the `cassandra-hadoop-util` from `janusgraph-cql` However, that makes no difference that I can tell. In my reproduction below, I put ```xml org.janusgraph cassandra-hadoop-util ``` in the `janusgraph-cql` dependency in my `pom.xml` and nothing changed. I assume that this is because the classes like `CQLHadoopScanRunner` are in `janusgraph-cql`, not in `cassandra-hadoop-util`. In my toy reproduction, I can make these warnings go away by adding dependencies on `janusgraph-hadoop` and `gremlin-server`. In my real application, adding `janusgraph-hadoop` causes a different problem (to do with Spring initialization). I can try to debug that if adding this dependency is the correct thing to do, but I'd like confirmation that this really is the correct approach. We're not using Hadoop to the best of my knowledge, so being forced to add the `janusgraph-hadoop` dependency would feel like an odd thing to do. Alternatively, we could ignore the warnings, or we could even repackage the `janusgraph-cql` JAR to exclude the classes with missing superclasses. But I'd like confirmation that this is going to be safe — I don't know whether they'd be used on some code path that we just haven't happened to hit in our testing so far. (Repackaging the JAR would also seem like an odd thing to do.) Any advice on how to proceed would be very welcome! Thanks. --- Full text of warnings: ``` [com.example.Repro.main()] WARN org.reflections8.Reflections - could not get type for name org.janusgraph.hadoop.scan.AbstractHadoopScanRunner from any class loader org.reflections8.ReflectionsException: could not get type for name org.janusgraph.hadoop.scan.AbstractHadoopScanRunner at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.(Reflections.java:158) at org.reflections8.Reflections.(Reflections.java:200) at org.reflections8.Reflections.(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.(JanusGraphTraverserUtil.java:35) at com.example.Repro.main(Repro.java:11) at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385) at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374) at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: org.janusgraph.hadoop.scan.AbstractHadoopScanRunner at org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader.loadClass(URLClassLoaderBuilder.java:181) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:410) ... 10 more [com.example.Repro.main()] WARN org.reflections8.Reflections - could not get type for name org.janusgraph.hadoop.formats.util.HadoopInputFormat from any class loader org.reflections8.ReflectionsException: could not get type for name org.janusgraph.hadoop.formats.util.HadoopInputFormat at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.(Reflections.java:158) at org.reflections8.Reflections.(Reflections.java:200) at org.reflections8.Reflections.(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.(JanusGraphTraverserUtil.java:35) at com.example.Repro.main(Repro.java:11) at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385) at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374) at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: org.janusgraph.hadoop.formats.util.HadoopInputFormat at org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader.loadClass(URLClassLoaderBuilder.java:181) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:410) ... 10 more [com.example.Repro.main()] WARN org.reflections8.Reflections - could not get type for name org.apache.tinkerpop.gremlin.server.GraphManager from any class loader org.reflections8.ReflectionsException: could not get type for name org.apache.tinkerpop.gremlin.server.GraphManager at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.(Reflections.java:158) at org.reflections8.Reflections.(Reflections.java:200) at org.reflections8.Reflections.(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.(JanusGraphTraverserUtil.java:35) at com.example.Repro.main(Repro.java:11) at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385) at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374) at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: org.apache.tinkerpop.gremlin.server.GraphManager at org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader.loadClass(URLClassLoaderBuilder.java:181) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:410) ... 10 more [com.example.Repro.main()] WARN org.reflections8.Reflections - could not get type for name org.janusgraph.hadoop.formats.util.AbstractBinaryInputFormat from any class loader org.reflections8.ReflectionsException: could not get type for name org.janusgraph.hadoop.formats.util.AbstractBinaryInputFormat at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.(Reflections.java:158) at org.reflections8.Reflections.(Reflections.java:200) at org.reflections8.Reflections.(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.(JanusGraphTraverserUtil.java:35) at com.example.Repro.main(Repro.java:11) at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385) at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374) at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: org.janusgraph.hadoop.formats.util.AbstractBinaryInputFormat at org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader.loadClass(URLClassLoaderBuilder.java:181) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:410) ... 10 more [com.example.Repro.main()] WARN org.reflections8.Reflections - could not get type for name org.janusgraph.hadoop.HadoopStoreManager from any class loader org.reflections8.ReflectionsException: could not get type for name org.janusgraph.hadoop.HadoopStoreManager at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.(Reflections.java:158) at org.reflections8.Reflections.(Reflections.java:200) at org.reflections8.Reflections.(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.(JanusGraphTraverserUtil.java:35) at com.example.Repro.main(Repro.java:11) at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385) at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374) at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: org.janusgraph.hadoop.HadoopStoreManager at org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader.loadClass(URLClassLoaderBuilder.java:181) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:410) ... 10 more [com.example.Repro.main()] WARN org.reflections8.Reflections - could not get type for name org.apache.hadoop.mapreduce.RecordReader from any class loader org.reflections8.ReflectionsException: could not get type for name org.apache.hadoop.mapreduce.RecordReader at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.(Reflections.java:158) at org.reflections8.Reflections.(Reflections.java:200) at org.reflections8.Reflections.(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.(JanusGraphTraverserUtil.java:35) at com.example.Repro.main(Repro.java:11) at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385) at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374) at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.RecordReader at org.codehaus.mojo.exec.URLClassLoaderBuilder$ExecJavaClassLoader.loadClass(URLClassLoaderBuilder.java:181) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:410) ... 10 more ``` --- Standalone reproduction: In `src/main/java/com/example/Repro.java` I have: ```java package com.example; import org.janusgraph.graphdb.util.JanusGraphTraverserUtil; public class Repro { public static void main(String[] args) { // Construct a new JanusGraphTraverserUtil to trigger its static initializer block. // Obviously we would never do this in real life — but the static initializer will be triggered in normal usage. // This is the simplest way to reproduce. JanusGraphTraverserUtil unused = new JanusGraphTraverserUtil(); } } ``` In `pom.xml` I have: ```xml 4.0.0 com.example repro 1.0-SNAPSHOT repro UTF-8 1.8 1.8 org.janusgraph janusgraph-core 1.0.0 org.janusgraph janusgraph-cql 1.0.0 org.slf4j slf4j-simple 1.7.36 maven-clean-plugin 3.1.0 maven-resources-plugin 3.0.2 maven-compiler-plugin 3.8.0 maven-jar-plugin 3.0.2 maven-install-plugin 2.5.2 ``` And then I do `mvn clean compile exec:java -Dexec.mainClass=com.example.Repro` to get the output shown above.

[dxtr-1-0]

I am also facing similar warnings with Hbase. Although i have verified that they do not cause issues in actual working of the code but still feels like this might be an issue. 2024-06-10 15:03:16.236 WARN o.r.Reflections:418 - could not get type for name org.janusgraph.hadoop.HadoopStoreManager from any class loader org.reflections8.ReflectionsException: could not get type for name org.janusgraph.hadoop.HadoopStoreManager at org.reflections8.ReflectionUtils.forName(ReflectionUtils.java:412) at org.reflections8.Reflections.expandSuperTypes(Reflections.java:421) at org.reflections8.Reflections.<init>(Reflections.java:158) at org.reflections8.Reflections.<init>(Reflections.java:200) at org.reflections8.Reflections.<init>(Reflections.java:173) at org.janusgraph.graphdb.util.JanusGraphTraverserUtil.<clinit>(JanusGraphTraverserUtil.java:35) at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphMultiQueryStep.registerTraverser(JanusGraphMultiQueryStep.java:119) at org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphMultiQueryStep.processNextStart(JanusGraphMultiQueryStep.java:112) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:155) at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.hasNext(ExpandableStepIterator.java:47) at org.apache.tinkerpop.gremlin.process.traversal.step.map.NoOpBarrierStep.processAllStarts(NoOpBarrierStep.java:67) at org.apache.tinkerpop.gremlin.process.traversal.step.map.NoOpBarrierStep.processNextStart(NoOpBarrierStep.java:56) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:155) at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55) at org.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:48) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:140)