search

oracle / opengrok

OpenGrok is a fast and usable source code search and cross reference engine, written in Java
http://oracle.github.io/opengrok/
Other
4.36k stars 747 forks source link

add Java version check #3655

Closed sahoode closed 3 years ago

sahoode commented 3 years ago

Describe the bug Indexing is failing

Enviorment : centos 6 tomcat10 java16

To Reproduce a restart of tomcat server gives below error:

30-Jun-2021 08:34:09.108 SEVERE [ForkJoinPool-1-worker-3] org.opengrok.suggest.SuggesterProjectData.initSearchCountMap Could not create ChronicleMap for field defs in directory /var/opengrok/data/suggester/92xx_scale-git , most popular completion disabled, if you are using JDK9+ make sure to specify: --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports java.base/sun.nio.ch=ALL-UNNAMED
    java.lang.NoClassDefFoundError: Could not initialize class net.openhft.chronicle.core.OS
        at net.openhft.chronicle.map.ChronicleMapBuilder.<init>(ChronicleMapBuilder.java:215)
        at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:241)
        at net.openhft.chronicle.map.ChronicleMap.of(ChronicleMap.java:72)
        at org.opengrok.suggest.popular.impl.chronicle.ChronicleMapAdapter.<init>(ChronicleMapAdapter.java:50)
        at org.opengrok.suggest.SuggesterProjectData.initSearchCountMap(SuggesterProjectData.java:313)
        at org.opengrok.suggest.SuggesterProjectData.init(SuggesterProjectData.java:161)
        at org.opengrok.suggest.Suggester.lambda$getInitRunnable$1(Suggester.java:229)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1414)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:295)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Expected behavior all projects are indexed

Additional context half of the project is indexed.

sahoode commented 3 years ago 
30-Jun-2021 08:33:58.917 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/etc/tomcat10/webapps/source.war]
30-Jun-2021 08:33:58.931 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Context] failed to set property [antiJARLocking] to [true]
30-Jun-2021 08:34:06.478 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
30-Jun-2021 08:34:06.674 INFO [main] org.opengrok.indexer.Metrics.<clinit> configuring PrometheusRegistry
30-Jun-2021 08:34:06.830 INFO [main] org.opengrok.web.WebappListener.contextInitialized Starting webapp with version 1.7.13 (0dc20a000a49f8f3533eed6ff3eb66ae2eb5ed86)
30-Jun-2021 08:34:06.831 INFO [main] org.opengrok.indexer.configuration.Configuration.read Reading configuration from /var/opengrok/etc/tomcat10/configuration.xml
30-Jun-2021 08:34:07.438 SEVERE [invalidate-repos-21] org.opengrok.indexer.util.Executor.exec Failed to read from process: svn
    java.io.IOException: An error occurred while parsing the xml output
        at org.opengrok.indexer.history.SubversionHistoryParser.processStream(SubversionHistoryParser.java:227)
        at org.opengrok.indexer.util.Executor.exec(Executor.java:226)
        at org.opengrok.indexer.history.SubversionHistoryParser.parse(SubversionHistoryParser.java:198)
        at org.opengrok.indexer.history.SubversionRepository.getHistory(SubversionRepository.java:289)
        at org.opengrok.indexer.history.SubversionRepository.determineCurrentVersion(SubversionRepository.java:409)
        at org.opengrok.indexer.history.RepositoryFactory.getRepository(RepositoryFactory.java:238)
        at org.opengrok.indexer.history.RepositoryFactory.getRepository(RepositoryFactory.java:155)
        at org.opengrok.indexer.history.RepositoryFactory.getRepository(RepositoryFactory.java:278)
        at org.opengrok.indexer.history.HistoryGuru.lambda$invalidateRepositories$5(HistoryGuru.java:830)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:831)
    Caused by: org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity.
        at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
        at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1471)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:942)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:562)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1504)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1925)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1560)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2761)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:541)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
        at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
        at org.opengrok.indexer.history.SubversionHistoryParser.processStream(SubversionHistoryParser.java:225)
        ... 13 more
30-Jun-2021 08:34:07.439 WARNING [invalidate-repos-21] org.opengrok.indexer.util.Executor.exec Non-zero exit status 1 from command [svn log --non-interactive --xml -v -l1 $'/var/opengrok/src/margot/c2c-git/cmn/protobuf-c-0.15/scripts@'] in directory /var/opengrok/src/margot/c2c-git/cmn/protobuf-c-0.15/scripts
30-Jun-2021 08:34:07.442 WARNING [invalidate-repos-21] org.opengrok.indexer.history.SubversionRepository.determineCurrentVersion cannot get current version info for /var/opengrok/src/margot/c2c-git/cmn/protobuf-c-0.15/scripts
30-Jun-2021 08:34:07.447 SEVERE [invalidate-repos-18] org.opengrok.indexer.util.Executor.exec Failed to read from process: svn
    java.io.IOException: An error occurred while parsing the xml output
        at org.opengrok.indexer.history.SubversionHistoryParser.processStream(SubversionHistoryParser.java:227)
        at org.opengrok.indexer.util.Executor.exec(Executor.java:226)
        at org.opengrok.indexer.history.SubversionHistoryParser.parse(SubversionHistoryParser.java:198)
        at org.opengrok.indexer.history.SubversionRepository.getHistory(SubversionRepository.java:289)
        at org.opengrok.indexer.history.SubversionRepository.determineCurrentVersion(SubversionRepository.java:409)
        at org.opengrok.indexer.history.RepositoryFactory.getRepository(RepositoryFactory.java:238)
        at org.opengrok.indexer.history.RepositoryFactory.getRepository(RepositoryFactory.java:155)
        at org.opengrok.indexer.history.RepositoryFactory.getRepository(RepositoryFactory.java:278)
        at org.opengrok.indexer.history.HistoryGuru.lambda$invalidateRepositories$5(HistoryGuru.java:830)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:831)
    Caused by: org.xml.sax.SAXParseException; lineNumber: 3; columnNumber: 1; XML document structures must start and end within the same entity.
        at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
        at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1471)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:942)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:562)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1504)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1925)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1560)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2761)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:541)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
        at java.xml/javax.xml.parsers.SAXParser.parse(SAXParser.java:197)
        at org.opengrok.indexer.history.SubversionHistoryParser.processStream(SubversionHistoryParser.java:225)
        ... 13 more
vladak commented 3 years ago

the log says 1.7.13 so updating the title accordingly.

vladak commented 3 years ago

There are 2 problems here:

vladak commented 3 years ago

Also, the topic claims indexing is failing however only logs from the web application are supplied.

ahornace commented 3 years ago

Could you also try with Java 11? I don't think we tested OpenGrok with Java 16 and it can cause some problems - especially with suggester (Chronicle Map).

sahoode commented 3 years ago

I downgraded my jdk to 11 Now I get errors while starting the tomcat. 30-Jun-2021 12:04:27.418 WARNING [main] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/etc/tomcat10/webapps/source/WEB-INF/lib/chronicle-core-2.20.80.jar] from classloader hierarchy java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:209) at org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65) at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49) at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:383) at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:318) at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:282) at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:233) at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262) at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104) at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:85) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5168) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:747) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1188) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2023) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1098) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:480) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1708) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:320) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:886) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:795) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:206) ... 49 more Caused by: java.nio.file.NoSuchFileException: /etc/tomcat10/webapps/source/WEB-INF/lib/chronicle-core-2.20.80.jar at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1764) at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1231) at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:732) at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:841) at java.base/java.util.zip.ZipFile.(ZipFile.java:247) at java.base/java.util.zip.ZipFile.(ZipFile.java:177) at java.base/java.util.jar.JarFile.(JarFile.java:348) ... 53 more 30-Jun-2021 12:04:27.418 WARNING [main] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/etc/tomcat10/webapps/source/WEB-INF/lib/annotations-19.0.0.jar] from classloader hierarchy java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:209) at org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65) at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49) at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:383) at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:318) at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:282) at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:233) at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262) at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104) at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:85) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5168) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:747) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1188) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2023) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1098) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:480) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1708) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:320) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:886) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:795) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:206) ... 49 more Caused by: java.nio.file.NoSuchFileException: /etc/tomcat10/webapps/source/WEB-INF/lib/annotations-19.0.0.jar at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1764) at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1231) at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:732) at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:841) at java.base/java.util.zip.ZipFile.(ZipFile.java:247) at java.base/java.util.zip.ZipFile.(ZipFile.java:177) at java.base/java.util.jar.JarFile.(JarFile.java:348) ... 53 more 30-Jun-2021 12:04:27.419 WARNING [main] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/etc/tomcat10/webapps/source/WEB-INF/lib/javax.el-3.0.0.jar] from classloader hierarchy java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:209) at org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65) at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49) at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:383) at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:318) at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:282) at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:233) at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262) at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104) at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:85) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5168) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:728) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:747) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1188) at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2023) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1098) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:480) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1708) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:320) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:948) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:886) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1398) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1388) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:921) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:437) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:934) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:795) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:206) ... 49 more Caused by: java.nio.file.NoSuchFileException: /etc/tomcat10/webapps/source/WEB-INF/lib/javax.el-3.0.0.jar at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1764) at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1231) at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:732) at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:841) at java.base/java.util.zip.ZipFile.(ZipFile.java:247) at java.base/java.util.zip.ZipFile.(ZipFile.java:177) at java.base/java.util.jar.JarFile.(JarFile.java:348) ... 53 more

Can I remove my source folder and try

ahornace commented 3 years ago

These logs are just warnings and are not relevant, OpenGrok should work fine. See: https://github.com/oracle/opengrok/issues/3390

sahoode commented 3 years ago

Thank you so much for the help team, And Now I see Whenever I restart my Opnegrok tomcat the UI resets back to some old data, then again i do a refresh and the new data comes up but same if I again do restart everything is gone and old data shows up. what I can send to do the analyis

ahornace commented 3 years ago

You should use -W /var/opengrok/etc/configuration.xml option when running indexer to actually preserve the configuration. Running the indexer with -U option sends the new config to the webapp but it does not actually store it on the disk.

sahoode commented 3 years ago

I am using java -verbose -jar /var/opengrok/dist/lib/opengrok.jar -R /var/opengrok/etc/tomcat10/configuration.xml -U http://localhost:8080/source

Am i using it wrong?

ahornace commented 3 years ago

Webapp has a hardcoded configuration path which is /var/opengrok/etc/configuration.xml by default. This configuration is loaded at every webapp startup. So if you are just using -R /var/opengrok/etc/tomcat10/configuration.xml -U http://localhost:8080/source, then the new configuration will just be uploaded to the running webapp.

You need to use -W /var/opengrok/etc/configuration.xml indexer option or webapp's REST API after the new configuration is uploaded to actually preserve it.

sahoode commented 3 years ago

When I use -W option it asks for source and data dirextory

java -verbose -jar /var/opengrok/dist/lib/opengrok.jar -W /var/opengrok/etc/tomcat10/configuration.xml -s /var/opengrok/src/margot -d /var/opengrok/data

ahornace commented 3 years ago

Because that information was previously contained in the readonly config provided with the -R option. Now you did not provide it so you need to convey that info via -d and -s.

sahoode commented 3 years ago

I am running this command java -verbose -jar /var/opengrok/dist/lib/opengrok.jar -W /var/opengrok/etc/tomcat10/configuration.xml -s /var/opengrok/src/margot/ -d /var/opengrok/data/

But I added one project which is not getting added even if it has the same location as the other projects.

And my opengrok shows

Screenshot 2021-07-13 at 6 55 58 AM

Please help.

vladak commented 3 years ago

But I added one project which is not getting added even if it has the same location as the other projects.

how did you add the project exactly ?

sahoode commented 3 years ago

I have added my repository content to /var/opengrok/src/margot/ and do git pull there. then run the indexer.

vladak commented 3 years ago

How do you run the indexer exactly ?

sahoode commented 3 years ago

When I first Installed my opengrok and when data folder is empty.

java -verbose -Djava.util.logging.config.file=/var/opengrok/etc/logging.properties \
    -jar /var/opengrok/dist/lib/opengrok.jar \
    -c /usr/local/bin/bin/ctags \
    -s /var/opengrok/src/margot -d /var/opengrok/data -H -P -S -G \
    -W /var/opengrok/etc/tomcat10/configuration.xml -U http://localhost:8080/source

and I have created a user for this p4build so i switch to p4build before running indexer.

And the daily I run a script which does the below job:

to update daily and the things mess up slowly.

sahoode commented 3 years ago

if you can suggest a clean way I can remove my data directory and start fresh indexing

vladak commented 3 years ago

There are several troubles with the way how the indexer is run periodically:

  1. missing feature options compared to the first run
  2. lack of signalling to the web app

Let me explain:

ad 1.: When the indexer is run for the first time, it has all these options that turn on some of the features, like projects (-P) or tags (-G) etc. This run also has the -W option that stores the configuration (along with information about repositories etc.) in a file. For example: for projects, the resulting XML in the configuration file will contain 

  <void property="projectsEnabled">
   <boolean>true</boolean>
  </void>

When the webapp loads this configuration on startup, it will make it enable the projects. Now, the periodic run of the indexer will overwrite the same configuration file. Because it is run without the -P option, the projects will be set to their default value, i.e. disabled (the above snippet will be missing in the file). When the web app gets redeployed (or the server restarted), it will load this new configuration and will therefore disable projects (or any other feature that is off by default and the options for it are missing in the periodic indexer run). Likewise for the other missing options.

ad 2: when the indexer is done, it needs to signal to the web app to refresh the index readers so that the searches can return fresh information. Also, the web app needs to receive meta information about repositories so that this is properly displayed on the index page. This can only be done when the indexer is run with the -U option. Otherwise the web app would have to be redeployed to read fresh configuration from disk.

In sum: usually, in the common case there is no reason to run the indexer with different set of option to the initial run (unless you want to follow the per project workflow).

sahoode commented 3 years ago

Ok got it now what is happening in the background thank you so much. So what should be the best indexer command to run for the periodic one.

vladak commented 3 years ago

So what should be the best indexer command to run for the periodic one.

the same as the initial one

vladak commented 3 years ago

Let's use this bug to add check to both indexer and web app for Java version. Anything outside allowed range will make them fail.

vladak commented 3 years ago

as discussed in the related PR the Java check is not needed, therefore closing.

sahoode commented 3 years ago

Thank you