"Raw" clicked and showed an error: java.lang.IllegalStateException: getOutputStream() has already been called for this response

[Ericwonne]

Describe the bug The Opengrok page is at large working well, but when clicking on the "Raw" link, which afterwards should give a page of code of plain text. Instead, it gives an error saying: getOutputStream() has already been called for this response

To Reproduce Simply click "Raw", other information: running "apache-tomcat-9.0.16/bin/version.sh" will have output: Using CATALINA_BASE: /home/Workspace/apache-tomcat-9.0.16 Using CATALINA_HOME: /home/Workspace/apache-tomcat-9.0.16 Using CATALINA_TMPDIR: /home/Workspace/apache-tomcat-9.0.16/temp Using JRE_HOME: /usr/java/latest/jre Using CLASSPATH: /home/Workspace/apache-tomcat-9.0.16/bin/bootstrap.jar:/home/Workspace/apache-tomcat-9.0.16/bin/tomcat-juli.jar Server version: Apache Tomcat/9.0.16 Server built: Feb 4 2019 16:30:29 UTC Server number: 9.0.16.0 OS Name: Linux OS Version: 3.12.39-47-default Architecture: amd64 JVM Version: 1.8.0_60-b27 JVM Vendor: Oracle Corporation

Expected behavior Plain text code.

Screenshots

Additional context Add any other context about the problem here.

[vladak]

What is your OpenGrok version ? Does this happen for all files or just for some ?

[vladak]

Is this just an error message or exception with stack trace ? (the screenshot did not make it)

[vladak]

Anyhow, the meat of raw JSP is this code:

86         OutputStream o = response.getOutputStream();
87         byte[] buffer = new byte[8192];
88         int nr;
89         while ((nr = in.read(buffer)) > 0) {
90             o.write(buffer, 0, nr);
91         }

and as https://stackoverflow.com/a/1776161/11582827 says, it is not a good idea to access the response output stream directly.

[Ericwonne]

However, this late reply indicates this yet unsolved error, trivial but bothering. Copy directly the stack trace below:

-----------------------------------Separator----------------------------------- There was an error! java.lang.IllegalStateException: getOutputStream() has already been called for this response

org.apache.jasper.JasperException: java.lang.IllegalStateException: getOutputStream() has already been called for this response at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:638) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:509) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:384) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:530) at org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:624) at org.apache.jsp.raw_jsp._jspService(raw_jsp.java:209) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.opengrok.web.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:42) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.opengrok.web.StatisticsFilter.doFilter(StatisticsFilter.java:59) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.opengrok.web.AuthorizationFilter.doFilter(AuthorizationFilter.java:98) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: getOutputStream() has already been called for this response at org.apache.catalina.connector.Response.getWriter(Response.java:582) at org.apache.catalina.connector.ResponseFacade.getWriter(ResponseFacade.java:227) at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:115) at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:108) at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:179) at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:120) at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:75) at org.apache.jsp.error_jsp._jspService(error_jsp.java:380) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476) ... 52 more

Last Index update Wed Feb 26 20:56:10 EST 2020

[Ericwonne]

I have multiple opengrok sites, and one of them works just fine - "Raw" clickable, no error showing. I assume it is a site-specific problem, however, I couldn't dig the cause out of it.

I set up my multiple sites this way: firstly duplicate each folder, name them slightly differently. And each duplicate uses a separate indexing command.

Me@~/home/workspace/opengrok> ls

data data-1 data-2 dist etc etc-1 etc-2 log src src-1 src-2

then in each src folder, there are soft links attached to each repo. And I use one single apache tomcat server to serve all the sites. Server version: Apache Tomcat/9.0.16

My version is opengrok-1.3.3. Could this "multiple" be the problem cause?

[idodeclare]

Could this "multiple" be the problem cause?

Unclear, but the stack does indicate that some error is being reported:

at org.apache.jsp.error_jsp._jspService(error_jsp.java:380)

but then the error.jsp is throwing that IllegalStateException when the error page tries to get a writer:

at org.apache.catalina.connector.Response.getWriter(Response.java:582)

Maybe catalina.out will have a hint?

[Ericwonne]

I read the catalina.out file, >7 megabytes, not knowing the significant message. I'd copy the lines after a restart of the apache tomcat server:

26-Feb-2020 22:15:28.939 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [45,691] milliseconds 26-Feb-2020 22:15:29.104 INFO [Thread-36146] org.opengrok.suggest.Suggester.shutdownAndAwaitTermination Suggester successfully initialized (took 2 seconds) 26-Feb-2020 22:15:29.131 INFO [Thread-36146] org.opengrok.web.api.v1.suggester.provider.service.impl.SuggesterServiceImpl.scheduleRebuild Scheduling suggester rebuild in PT1H44M30.896S 26-Feb-2020 22:18:59.368 WARNING [http-nio-17845-exec-5] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:00.260 WARNING [http-nio-17845-exec-1] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:00.490 WARNING [http-nio-17845-exec-4] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:00.496 WARNING [http-nio-17845-exec-10] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:41.294 WARNING [http-nio-17845-exec-1] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:42.457 WARNING [http-nio-17845-exec-6] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:42.717 WARNING [http-nio-17845-exec-10] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:19:42.717 WARNING [http-nio-17845-exec-2] org.opengrok.suggest.SuggesterProjectData.lookup No WFST for field defs in /Workspace/opengrok/data/suggester/pxe-unity-finder-counter-goshawk 26-Feb-2020 22:22:25.868 SEVERE [http-nio-17845-exec-10] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:27.450 INFO [http-nio-17845-exec-10] org.opengrok.suggest.Suggester.rebuild Rebuilding the following suggesters: [pxe-unity-finder-counter] 26-Feb-2020 22:22:34.524 SEVERE [http-nio-17845-exec-9] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:37.171 SEVERE [http-nio-17845-exec-2] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:41.979 SEVERE [http-nio-17845-exec-4] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:42.538 SEVERE [http-nio-17845-exec-7] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:44.668 SEVERE [http-nio-17845-exec-8] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:46.625 SEVERE [http-nio-17845-exec-6] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:46.913 SEVERE [http-nio-17845-exec-1] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:48.654 SEVERE [http-nio-17845-exec-5] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:50.385 SEVERE [http-nio-17845-exec-3] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:22:53.107 SEVERE [http-nio-17845-exec-11] org.opengrok.indexer.history.AccuRevRepository.getAccuRevInfo Not logged into AccuRev server 26-Feb-2020 22:23:14.265 INFO [http-nio-17845-exec-10] org.opengrok.suggest.Suggester.shutdownAndAwaitTermination Suggesters for [pxe-unity-finder-counter] were successfully rebuilt (took 46 seconds) 26-Feb-2020 22:23:14.266 INFO [http-nio-17845-exec-11] org.opengrok.suggest.Suggester.rebuild Rebuilding the following suggesters: [pxe-unity-finder-counter]

[Ericwonne]

I tried to index from command, which then gives a long output,

firstly gives like

SEVERE: Not logged into AccuRev server 22:16:27 SEVERE: Not logged into AccuRev server 22:16:29 SEVERE: Not logged into AccuRev server 22:16:30 SEVERE: Not logged into AccuRev server 22:16:31 SEVERE: Not logged into AccuRev server 22:16:33 SEVERE: Not logged into AccuRev server 22:16:34 SEVERE: Not logged into AccuRev server 22:16:36 SEVERE: Not logged into AccuRev server

Then 22:21:56 WARNING: ERROR addFile(): /Workspace/opengrok/src/pxematos/source/case/test/dev/sdnas/restkk/broadcom_test.gz

Then SEVERE: Problem updating index database in directory /Workspace/opengrok/data/index/pxematos

[idodeclare]

Strange, that is the second recent issue (see #3052) where AccuRev comes up.

Do you have an accurev binary installed?

If you're running version 1.3.8 or later, you can try --disableRepository accurev.

[idodeclare]

Regarding catalina.out, I meant to look at recent entries. You might also look to files named catalina.\<ISO-DATE>.log which are segmented but otherwise identical to catalina.out.

I.e. trigger the problem, and then check e.g. catalina.2020-02-26.log.

[Ericwonne]

I don't know if AccuRev is being installed, what I know is I never manually install it myself. The AccuRev problem is not coincidental, it almost happens every time when I start (re)indexing, but never caused the "Raw unclickable problem".

[vladak]

My version is opengrok-1.3.3. Could this "multiple" be the problem cause?

The problem reported in this issue was fixed in 1.3.7.

[Ericwonne]

There is one little difference between the normally-behaved opengrok site and the malfunctioning one, in regard to the "Raw unclickable problem".

When clicking the raw button which gives an error, the URL becomes ending with _aaa/bbb/ccc/ddd.t.fileext?r=f0b6cfe1, and if remove the r=f0b6cfe1, the raw file will show normally.

@vladak By the way,

My version is opengrok-1.3.3. Could this "multiple" be the problem cause?

The problem reported in this issue was fixed in 1.3.7.

So should I simply upgrade my opengrok to a higher version and the problem will be solved? However, anything specific to focus on? Thank you so much for your assistance.

[Ericwonne]

Would there be anything to take care with when upgrading to a higher version of opengrok? How do I deal with the src folder?

[vladak]

Would there be anything to take care with when upgrading to a higher version of opengrok? How do I deal with the src folder?

In general take a look at https://github.com/oracle/opengrok/releases and go through the individual releases from the one you are currently running to the one you are going to upgrade to and see if there is anything of interest.

In this case (1.3.3 -> 1.3.7) you do not need to reindex from scratch so just install the new version and deploy the web application.

[vladak]

There is one little difference between the normally-behaved opengrok site and the malfunctioning one, in regard to the "Raw unclickable problem".

When clicking the raw button which gives an error, the URL becomes ending with _aaa/bbb/ccc/ddd.t.fileext?r=f0b6cfe1, and if remove the r=f0b6cfe1, the raw file will show normally.

Not sure how much does it contribute to the problem, the revision number was added to the URL in #1743 and is therefore fully expected. Anyhow, upgrading to the latest greatest should resolve the getOutputStream() problem.

[Ericwonne]

Thank you but I have to ask that the "upgrading" means that I download the latest version, untar, then copy each file to the location accordingly and then reindex?

[vladak]

Thank you but I have to ask that the "upgrading" means that I download the latest version, untar, then copy each file to the location accordingly and then reindex?

Pretty much. Here's a my workflow for minor (i.e. no reindex from scratch needed) updates (it is partly specific to Solaris and our deployment location, you get the idea I hope):

• NOTE: this is best to do if the reindex is not running. If you want to be certain, comment out the entry that does the syncing/indexing from crontab and then re-enable it after this process is complete.
• download the tar ball from https://github.com/OpenGrok/opengrok/releases into /var/tmp
• make the current configuration persistent (this is done so that information about last indexed revisions of repositories are kept current to have them properly displayed on the index page).

  zfs snapshot data/opengrok/etc@`date '+%y-%m-%dT%H:%M:%S'`
  LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8" /opengrok/dist/bin/venv/bin/opengrok-projadm --jar /opengrok/dist/lib/opengrok.jar \
    -c /opengrok/dist/bin/venv/bin/opengrok-config-merge -b /opengrok -R /opengrok/etc/readonly_configuration.xml -r

• extract the OpenGrok tarball (substitute the X.Y with the version you would like to extract) and update the Python OpenGrok tools:

  gtar -C /opengrok/dist --strip-components=1 -xzf /var/tmp/opengrok-X.Y.tar.gz
  /opengrok/dist/bin/venv/bin/pip \
     install /opengrok/dist/tools/opengrok-tools.tar.gz

• deploy: copy the source.war file to Tomcat directory (passing the configuration location is necessary as there is no /var/opengrok/etc/configuration.xml file which is the default location. If this is not done, the webapp will break)

  /opengrok/dist/bin/venv/bin/opengrok-deploy -c /opengrok/etc/configuration.xml \
     /opengrok/dist/lib/source.war /var/tomcat8/webapps/source.war

• check the state of the server (the redeploy takes several seconds as all repositories are scanned with their respective SCM commands)

  svcs -p tomcat8

• Check the logs for exceptions and messages with the SEVERE log level

  tail -f /var/tomcat8/logs/catalina.out

• go to web and check for any signs of trouble (UI functionality like search, directory browsing, history view, annotations, listings of projects should match the authorization etc.)

You can see that reindexing is actually not needed.