search

oracle / opengrok

OpenGrok is a fast and usable source code search and cross reference engine, written in Java
http://oracle.github.io/opengrok/
Other
4.36k stars 747 forks source link

NoSuchMethodError exception java.nio.MappedByteBuffer.position in ELF analyzer #2815

Closed Conan-Kudo closed 5 years ago

Conan-Kudo commented 5 years ago

I'm trying to run OpenGrok 1.2.14 on Ubuntu 16.04 on Java 8 using the system provided Java 8 and Tomcat 8 stack.

When running the indexer, I got the following tracebacks:

$ sudo opengrok-indexer -J=-Djava.util.logging.config.file=/var/opengrok/logging.properties -a /opt/opengrok/lib/opengrok.jar -- -s /var/opengrok/src -d /var/opengrok/data -H -P -S -G -W /var/opengrok/etc/configuration.xml -c /usr/bin/universal-ctags       
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "clojure" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "rust" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "pascal" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "clojure" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "rust" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "pascal" already defined
12:59:53 WARNING: Could not read /var/opengrok/src/device/platform/tests/vendor
13:00:06 WARNING: Error from ctags: universal-ctags: Warning: ignoring null tag in /var/opengrok/src/device/platform/web/UI/js/bootstrap.min.js(line: 6)
13:00:12 SEVERE: 2739 successes (75.3%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
        at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        ... 8 more
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.<init>(ELFAnalyzer.java:229)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
        at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
        at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
        ... 2 more

13:00:12 WARNING: 3637 failures (100.0%) while parallel-indexing
vladak commented 5 years ago

The code in question in opengrok-indexer/src/main/java/org/opengrok/indexer/analysis/executables/ELFAnalyzer.java:

106      public String parseELF(FileChannel fch) throws IOException {
107          MappedByteBuffer fmap = fch.map(FileChannel.MapMode.READ_ONLY, 0, fch.size());
108          ELFHeader eh = new ELFHeader(fmap);
...
185      private static class ELFHeader {
...
211          ELFHeader(MappedByteBuffer fmap) throws IllegalArgumentException {
212              if (fmap.get(ELFIdentification.EI_MAG0.value()) != 0x7f ||
213                  fmap.get(ELFIdentification.EI_MAG1.value()) != 'E' ||
214                  fmap.get(ELFIdentification.EI_MAG2.value()) != 'L' ||
215                  fmap.get(ELFIdentification.EI_MAG3.value()) != 'F') {
216                  throw new IllegalArgumentException("Not an ELF file");
217              }
218  
219              ei_class = EI_Class.valueOf(fmap.get(ELFIdentification.EI_CLASS.value()));
...
229              fmap.position(ELFIdentification.EI_NIDENT.value());

So, the MappedByteBuffer instance is present (certainly other methods from there were called, otherwise an exception would be thrown earlier), just somehow the position() method is not.

vladak commented 5 years ago

Actually, what is "system provided Java 8" exactly ?

Conan-Kudo commented 5 years ago

I'm using the openjdk-8-jdk-headless package from Ubuntu for my Java 8 runtime environment.

vladak commented 5 years ago

Just to be sure, could you run the indexer with -J=-verbose:class, using the output determine from which file it loads java.nio.Buffer, extract the Buffer.class from that jar file and run javap on it to see if it contains the position() methods ?

vladak commented 5 years ago

Also, I wonder how many ELF files do you have in your projects and whether all of the suffer from this problem.

Conan-Kudo commented 5 years ago

verbose output:

Jun 18, 2019 3:40:06 PM org.opengrok.indexer.index.IndexDatabase indexParallel
SEVERE: 323 successes (36.0%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
        at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        ... 8 more
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.<init>(ELFAnalyzer.java:229)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
        at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
        at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        ... 4 more

Jun 18, 2019 3:40:06 PM org.opengrok.indexer.index.IndexDatabase indexParallel
WARNING: 898 failures (100.0%) while parallel-indexing

There's only one jar file opengrok.jar, but I popped it open and ran javap on it:

root@opengrok-stg:/opt/opengrok/lib/test/org/opengrok/indexer/analysis/executables# javap ELFAnalyzer.class 
Compiled from "ELFAnalyzer.java"
public class org.opengrok.indexer.analysis.executables.ELFAnalyzer extends org.opengrok.indexer.analysis.FileAnalyzer {
  protected org.opengrok.indexer.analysis.executables.ELFAnalyzer(org.opengrok.indexer.analysis.AnalyzerFactory);
  protected int getSpecializedVersionNo();
  public void analyze(org.apache.lucene.document.Document, org.opengrok.indexer.analysis.StreamSource, java.io.Writer) throws java.io.IOException;
  public java.lang.String parseELF(java.nio.channels.FileChannel) throws java.io.IOException;
  static {};
}
Conan-Kudo commented 5 years ago

Also, I wonder how many ELF files do you have in your projects and whether all of the suffer from this problem.

ELF files do show up in a fair number of projects...

vladak commented 5 years ago

For reference on my laptop with Ubuntu 18.04 and Oracle Java 8 (build 1.8.0_181-b13) I stepped through the ELFHeader constructor in debugger (IDEA) past the call to position() without any problem.

vladak commented 5 years ago

The verbose output should look like this:

$ java -verbose:class -jar ~/opengrok-git-vladak/opengrok-indexer/target/opengrok-1.2.12.jar 2>&1 | head
[Opened /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.Object from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.io.Serializable from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.Comparable from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.CharSequence from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.String from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.reflect.AnnotatedElement from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.reflect.GenericDeclaration from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.reflect.Type from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.Class from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]

i.e. the -verbose:class option needs to be passed to java, not as option to the indexer.

vladak commented 5 years ago

Hence, your indexer line should look like this:

sudo opengrok-indexer -J=-verbose:class -J=-Djava.util.logging.config.file=/var/opengrok/logging.properties -a /opt/opengrok/lib/opengrok.jar -- -s /var/opengrok/src -d /var/opengrok/data -H -P -S -G -W /var/opengrok/etc/configuration.xml -c /usr/bin/universal-ctags
Conan-Kudo commented 5 years ago

So then the output is this?

[Loaded java.io.RandomAccessFile from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader from file:/opt/opengrok/lib/opengrok.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFIdentification from file:/opt/opengrok/lib/opengrok.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$EI_Class from file:/opt/opengrok/lib/opengrok.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$EI_Data from file:/opt/opengrok/lib/opengrok.jar]
[Loaded java.io.RandomAccessFile$1 from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal$Form from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal$1 from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal$2 from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.io.PrintWriter from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.Throwable$PrintStreamOrWriter from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.Throwable$WrappedPrintWriter from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.util.IdentityHashMap$KeySet from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
16:15:58 SEVERE: 143 successes (40.9%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
        at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        ... 8 more
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.<init>(ELFAnalyzer.java:229)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
        at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
        at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
        ... 2 more

16:15:58 WARNING: 350 failures (100.0%) while parallel-indexing
Conan-Kudo commented 5 years ago

From the rt.jar mentioned in the log:

root@opengrok-stg:~/foo# javap java/nio/Buffer.class 
Compiled from "Buffer.java"
public abstract class java.nio.Buffer {
  static final int SPLITERATOR_CHARACTERISTICS;
  long address;
  java.nio.Buffer(int, int, int, int);
  public final int capacity();
  public final int position();
  public final java.nio.Buffer position(int);
  public final int limit();
  public final java.nio.Buffer limit(int);
  public final java.nio.Buffer mark();
  public final java.nio.Buffer reset();
  public final java.nio.Buffer clear();
  public final java.nio.Buffer flip();
  public final java.nio.Buffer rewind();
  public final int remaining();
  public final boolean hasRemaining();
  public abstract boolean isReadOnly();
  public abstract boolean hasArray();
  public abstract java.lang.Object array();
  public abstract int arrayOffset();
  public abstract boolean isDirect();
  final int nextGetIndex();
  final int nextGetIndex(int);
  final int nextPutIndex();
  final int nextPutIndex(int);
  final int checkIndex(int);
  final int checkIndex(int, int);
  final int markValue();
  final void truncate();
  final void discardMark();
  static void checkBounds(int, int, int);
}
Conan-Kudo commented 5 years ago

My OpenJDK version from Ubuntu:

$ dpkg -l | grep openjdk
ii  openjdk-8-jdk-headless:amd64        8u212-b03-0ubuntu1.16.04.1                 amd64        OpenJDK Development Kit (JDK) (headless)
ii  openjdk-8-jre-headless:amd64        8u212-b03-0ubuntu1.16.04.1                 amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
vladak commented 5 years ago

Could you post the complete output w.r.t. the Loaded lines ?

Conan-Kudo commented 5 years ago

Attached: opengrok-index-run.txt

vladak commented 5 years ago

Hm, I don't see anything wrong there.

Conan-Kudo commented 5 years ago

@vladak Have you tried running 1.2.14 with Ubuntu's OpenJDK?

vladak commented 5 years ago

Nope. The Travis builds run with xenial+OpenJdk8 tho.

Dne st 19. 6. 2019 15:34 uživatel Neal Gompa (ニール・ゴンパ) < notifications@github.com> napsal:

@vladak https://github.com/vladak Have you tried running 1.2.14 with Ubuntu's OpenJDK?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/oracle/opengrok/issues/2815?email_source=notifications&email_token=AAWMMDD2IT7SOJDFXZ7RBQDP3IYW7A5CNFSM4HZCCVMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYB4F6Y#issuecomment-503563003, or mute the thread https://github.com/notifications/unsubscribe-auth/AAWMMDB3CGKZQP6CXBVE2PLP3IYW7ANCNFSM4HZCCVMA .

IamTHEvilONE commented 5 years ago

Is there any additional data I can provide on this, as I'm seeing something similar? (referenced from #2122)

CentOS Linux release 7.6.1810 (Core) opengrok-1.2.14 java-1.8.0-openjdk-headless-1.8.0.212.b04-0.el7_6.x86_64 java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64

[Loaded java.nio.MappedByteBuffer from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/lib/rt.jar]

vladak commented 5 years ago

Maybe you could try to find out if this is data driven, i.e. try to reduce the indexed data to the point when the issue is still happening. I wonder if this happens with just a single ELF file under source root.

IamTHEvilONE commented 5 years ago

So I previously had 10-12 different source codes in my src folder. Cut out all but one to reduce the possible data.

If I ignore any line from std out that's just loading code in verbose mode: this is standard out"

18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined 18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined 18:17:46 SEVERE: 1813 successes (40.4%) after aborting parallel-indexing java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006) at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229) at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496) at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoSuchMethodError at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598) at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005) ... 8 more Caused by: java.lang.NoSuchMethodError at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598) at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735) at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496) at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182) at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer; at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.(ELFAnalyzer.java:229) at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108) at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93) at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586) at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754) at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193) at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070) at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058) ... 2 more 18:17:46 WARNING: 4487 failures (100.0%) while parallel-indexing

18:17:52 SEVERE: ERROR addFile(): org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:680) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:694) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1613) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235) at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:780) at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193) at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070) at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

18:17:52 WARNING: An error occurred while finishing writer and completer

18:17:52 WARNING: An error occurred while finishing writer and completer java.util.ConcurrentModificationException at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1561) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) at java.util.stream.AbstractTask.compute(AbstractTask.java:316) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.opengrok.indexer.index.PendingFileCompleter.completeRenamings(PendingFileCompleter.java:195) at org.opengrok.indexer.index.PendingFileCompleter.complete(PendingFileCompleter.java:171) at org.opengrok.indexer.index.IndexDatabase.finishWriting(IndexDatabase.java:1660) at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:519) at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

18:17:52 SEVERE: Failed with unexpected RuntimeException java.util.ConcurrentModificationException at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1561) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) at java.util.stream.AbstractTask.compute(AbstractTask.java:316) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.opengrok.indexer.index.PendingFileCompleter.completeRenamings(PendingFileCompleter.java:195) at org.opengrok.indexer.index.PendingFileCompleter.complete(PendingFileCompleter.java:171) at org.opengrok.indexer.index.IndexDatabase.finishWriting(IndexDatabase.java:1660) at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:519) at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

18:17:52 SEVERE: Problem updating index database in directory /usr/opengrok/data-2/index/index-test: java.util.ConcurrentModificationException at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1561) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) at java.util.stream.AbstractTask.compute(AbstractTask.java:316) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734) at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.opengrok.indexer.index.PendingFileCompleter.completeRenamings(PendingFileCompleter.java:195) at org.opengrok.indexer.index.PendingFileCompleter.complete(PendingFileCompleter.java:171) at org.opengrok.indexer.index.IndexDatabase.finishWriting(IndexDatabase.java:1660) at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:519) at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

I'll see if I can scrub and attach the opengrok0.0.log

carbonrobin commented 5 years ago

I'm experiencing this problem also. Searching around shows other applications experiencing the problem on Java 8 when built with Java 9 or greater. I'm new to Opengrok, but did I see the releases are built using Java 11? Hopefully what I found helps.

Source: [https://github.com/lmdbjava/lmdbjava/issues/116]

Java 9 introduces overridden methods with covariant return types for the following methods in java.nio.ByteBuffer that are used by the driver

In Java 9 they all now return ByteBuffer, whereas the methods they override return Buffer, resulting in exceptions like this when executing on Java 8 and lower

This is because the generated byte code includes the static return type of the method, which is not found on Java 8 and lower because the overloaded methods with covariant return types don't exist.

The solution is to cast ByteBuffer instances to Buffer before calling the method.

carbonrobin commented 5 years ago

And it looks like a better option would be to add the flag --release 8 to javac when building.

Source: https://github.com/plasma-umass/doppio/issues/497 Source: https://github.com/eclipse/jetty.project/issues/3244 Source: https://www.reddit.com/r/androiddev/comments/ah194v/javaniobytebuffer_java_9_compiler_and_android/

vladak commented 5 years ago

This might be it. The deploy stage in Travis basically builds the whole distribution from scratch (which is another problem of its own) using JDK 11 (e.g. https://travis-ci.org/oracle/opengrok/jobs/548094045 for 1.2.15).

Curiously, Maven's has the compileTarget property set to 1.8.

vladak commented 5 years ago

I changed the deploy build in cset 54bbf931b4f9072cdb2de9297b85cdcc78ee09c2, will trigger new release.

vladak commented 5 years ago

relevant comment on the setting the compile target: https://github.com/plasma-umass/doppio/issues/497#issuecomment-353157781

vladak commented 5 years ago

It seems the ultimate cure for this problem is to use --release 8 as explained on https://github.com/eclipse/jetty.project/issues/3244#issuecomment-495322586 , however as I understand it one needs JDK 9+ for that.

vladak commented 5 years ago

@Conan-Kudo , @IamTHEvilONE, @carbonrobin once 1.2.16 is out please give it a try.

IamTHEvilONE commented 5 years ago

I have no parallel-indexing errors on 1.2.16 on a single source code repo, nor MappedByteBuffer errors.

Testing with my 12+ code bases I'd index.

Conan-Kudo commented 5 years ago

It seems to be okay for me so far...

vladak commented 5 years ago

Cool, closing then. Many thanks @carbonrobin !