search

IQSS / dataverse

Open source research data repository software
http://dataverse.org
Other
873 stars 484 forks source link

Better Logging for Malformed Requests #4204

Closed kcondon closed 6 years ago

kcondon commented 6 years ago

Server logs in production are growing so we should review what is being written and root cause. One particular source of errors is the HUIT WhiteHat security scans that generate lots of failed/malformed searches. Our app throws an outsized amount of stack traces for a single malformed search.

pdurbin commented 6 years ago

Back in #1212 we decided what do display to end users when they make a search syntax error. I wrote "you should see different errors/technical details if you search for ":" vs. searching for "foo:bar"" and @kcondon replied with "OK technical details are hidden behind link". Apparently, this behavior changed without us realizing it because as of 4.8.3 we no longer show "Search Syntax Error" like we used to. Here are some screenshots of the current behavior:

A search for :

screen shot 2017-11-22 at 11 38 58 am

A search for foo:bar

screen shot 2017-11-22 at 11 38 37 am

In order to keep this issue small, I assume we aren't trying to bring back the user experience from #1212 which said stuff like "Search Syntax Error: undefined field foo":

A search for foo:bar

4caea992-7bd0-11e4-8cec-92520140c06d

That is to say, my understanding is this issue is only about not spamming the logs on syntax errors. If we want to revisit the user experience when a syntax error is made, we'll create a new issue for that.

pdurbin commented 6 years ago

I made pull request #4303 and am putting it into code review at https://waffle.io/IQSS/dataverse

landreev commented 6 years ago

(sorry, I just closed this issue by accident...)

landreev commented 6 years ago

@pdurbin, I just saw your PR - I'm not sure that's going to fix the problem at hand. The PR just changes a log level from .info to .fine in 3 places. What's polluting the logs is an UNCAUGHT exception thrown somewhere by a bad search query. See the exception stack trace from the log files, below. (so, once again, it's not a single log entry - it's a giant stack trace for every bad query). Also note that it may not even be coming from our search/index/solr code. I haven't looked closely, but a bad query like this may be causing an exception way before it gets to the search api; it may still be in the incoming request parser.

landreev commented 6 years ago

this is a stack trace example; rather, just the top part of it, I think. look at the logs on dvn-60ox-app-2.lib.harvard.edu; that's the prod. server that gets hit by these scans.

[2017-11-27T05:45:11.457-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.web.core] [tid: _ThreadID=95 _ThreadName=jk-connector(4)] [timeMi llis: 1511779511457] [levelValue: 900] [[ Servlet.service() for servlet Faces Servlet threw exception java.net.URISyntaxException: Illegal character in query at index 25: /500.xhtml?q=&types=files"&sort=dateSort&order=asc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:577) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:505) at org.apache.catalina.core.StandardHostValve.dispatchToErrorPage(StandardHostValve.java:699) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:380) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:334) at org.apache.catalina.core.StandardHostValve.postInvoke(StandardHostValve.java:232) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:735) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)

... etc. etc. etc.

landreev commented 6 years ago

P.S. So, this may not be specific to bad searches - but to bad/malformed queries in general. The above stack trace was from server.log_2017-11-27T05-55-14 on dvn-60ox-app-2.

djbrooke commented 6 years ago

I mentioned in Sprint Planning today that my main concern here is that these logs are taking up considerable space. I'd like to focus on getting those log files to a reasonable size so that we don't have to buy more storage. 😧

matthew-a-dunlap commented 6 years ago

I don't have access to the production servers to reach these logs and see the types of malformed searches being used. Can someone help me get them @landreev @kcondon ?

matthew-a-dunlap commented 6 years ago

There has been a lot of discussion and investigating into this today, mostly by @landreev . It has taken a fair bit of digging to recreate the actual urls that are causing the log noise as all the log files seem to print out urls strings that have been altered ("sanitized") in some way. Furthermore, attempting to recreate these urls in the browser also fails because browsers do their own sanitization before sending things off. This curl is able to trigger the logs we are experiencing (and curl matches up with the way these scans are being run, anyways)

curl 'https://dvn-build.hmdc.harvard.edu/dataverse/root;jsessionid=5abc0618cbf625c1f79439a431fd?q=&types=files"&sort=nameSort&order=desc'

As of now, we aren't entirely sure how these searches are flowing through the codebase and what specifically is creating all these logs. We have some hunches that will be tested tomorrow.

landreev commented 6 years ago

Thank you @matthew-a-dunlap for the summary of the last day's research. A few additional things:

  1. The "noise" (exception stack traces) is DEFINITELY NOT coming from our code. It's not caused by our search service. I went back and forth on this yesterday, sorry for the confusion.

  2. To reiterate, from Matthew's comment above, use curl to reproduce the offending queries (such as the one with the double quote above, 'types=files"'). If you do it from the browser, it will encode it for you as files%22 - which does not cause a 500.

  3. So no, the noise is not a result of the scan DOSing us with extra load. This is perfectly reproducible under normal conditions, one request at a time (see 2. above)

  4. It's just the bad URL (such as the URL in Matthew's example, with the unencoded double quote) that is causing the page to fail to render, with a 500 and long stack traces. Some extra confusion: this query happens to be a bad, malformed search query too. So when the search fragment sends it to SOLR, it gets back error messages. These are logged compactly (as .INFO; that @pdurbin has since changed to .FINE). However, this is NOT what's causing the mess. Commenting out the searchservice call, and completely removing search-include-fragment from the dataverse page still results in the same noisy failure.

  5. The failure appears to happen at the very end, once the page has gathered all the needed content (as in, the SearchIncludeFragment is properly initialized and the searches are run, see 4. above). At the very last moment, Faces tries to do something with the URL, and realizes - oh crap, the url is bad!

  6. Will add the full stack trace, below. Once again, none of the classes mentioned are ours - it's all Grizzly/faces/etc. Not sure what to do with all this.

landreev commented 6 years ago

Below is the full stack trace caused by 

curl  'http://localhost:8080/dataverse/root;jsessionid=5abc0618cbf625c1f79439a431fd?q=&types=files"&sort=nameSort&order=desc'

[2017-11-29T20:12:21.442-0500] [glassfish 4.1] [SEVERE] [] [javax.enterprise.resource.webcontainer.jsf.application] [tid: _ThreadID=33 _ThreadName=http-listener-1(4)] [timeMillis: 1512004341442] [levelValue: 1000] [[ Error Rendering View[/dataverse.xhtml] java.lang.IllegalArgumentException: [/dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc] is not a valid URL fragment. Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder] at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:89) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:41) at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:268) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:188) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.URISyntaxException: Illegal character in query at index 31: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) ... 68 more ]]

[2017-11-29T20:12:21.443-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.web.core] [tid: _ThreadID=33 _ThreadName=http-listener-1(4)] [timeMillis: 1512004341443] [levelValue: 900] [[ Servlet.service() for servlet Faces Servlet threw exception java.net.URISyntaxException: Illegal character in query at index 31: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:41) at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:268) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:188) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) ]]

[2017-11-29T20:12:21.443-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=33 _ThreadName=http-listener-1(4)] [timeMillis: 1512004341443] [levelValue: 900] [[ StandardWrapperValve[default]: Servlet.service() for servlet default threw exception java.net.URISyntaxException: Illegal character in query at index 31: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:575) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:378) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteResultHandler.handleResult(HttpRewriteResultHandler.java:41) at org.ocpsoft.rewrite.servlet.RewriteFilter.rewrite(RewriteFilter.java:268) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:188) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:316) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:160) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) ]]

[2017-11-29T20:12:21.477-0500] [glassfish 4.1] [SEVERE] [] [javax.enterprise.resource.webcontainer.jsf.application] [tid: _ThreadID=33 _ThreadName=http-listener-1(4)] [timeMillis: 1512004341477] [levelValue: 1000] [[ Error Rendering View[/500.xhtml] java.lang.IllegalArgumentException: [/500.xhtml?q=&types=files"&sort=nameSort&order=desc] is not a valid URL fragment. Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder] at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:89) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:577) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:505) at org.apache.catalina.core.StandardHostValve.dispatchToErrorPage(StandardHostValve.java:699) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:380) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:334) at org.apache.catalina.core.StandardHostValve.postInvoke(StandardHostValve.java:232) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:735) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.URISyntaxException: Illegal character in query at index 25: /500.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) ... 61 more ]]

[2017-11-29T20:12:21.478-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.web.core] [tid: _ThreadID=33 _ThreadName=http-listener-1(4)] [timeMillis: 1512004341478] [levelValue: 900] [[ Servlet.service() for servlet Faces Servlet threw exception java.net.URISyntaxException: Illegal character in query at index 25: /500.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:577) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:505) at org.apache.catalina.core.StandardHostValve.dispatchToErrorPage(StandardHostValve.java:699) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:380) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:334) at org.apache.catalina.core.StandardHostValve.postInvoke(StandardHostValve.java:232) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:735) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) ]]

[2017-11-29T20:12:21.478-0500] [glassfish 4.1] [WARNING] [] [javax.enterprise.web] [tid: _ThreadID=33 _ThreadName=http-listener-1(4)] [timeMillis: 1512004341478] [levelValue: 900] [[ org.apache.catalina.core.StandardHostValve@1afcae9d: Exception Processing ErrorPage[errorCode=500, location=/500.xhtml] javax.servlet.ServletException: [/500.xhtml?q=&types=files"&sort=nameSort&order=desc] is not a valid URL fragment. Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder] at javax.faces.webapp.FacesServlet.service(FacesServlet.java:659) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:577) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:505) at org.apache.catalina.core.StandardHostValve.dispatchToErrorPage(StandardHostValve.java:699) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:380) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:334) at org.apache.catalina.core.StandardHostValve.postInvoke(StandardHostValve.java:232) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:735) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: [/500.xhtml?q=&types=files"&sort=nameSort&order=desc] is not a valid URL fragment. Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder] at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:89) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) ... 39 more Caused by: java.net.URISyntaxException: Illegal character in query at index 25: /500.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) ... 61 more ----- Root Cause ----- java.lang.IllegalArgumentException: [/500.xhtml?q=&types=files"&sort=nameSort&order=desc] is not a valid URL fragment. Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder] at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:89) at org.ocpsoft.rewrite.servlet.impl.HttpRewriteWrappedResponse.encodeURL(HttpRewriteWrappedResponse.java:390) at com.sun.faces.context.ExternalContextImpl.encodeActionURL(ExternalContextImpl.java:604) at javax.faces.context.ExternalContextWrapper.encodeActionURL(ExternalContextWrapper.java:108) at com.sun.faces.renderkit.html_basic.FormRenderer.getActionStr(FormRenderer.java:253) at com.sun.faces.renderkit.html_basic.FormRenderer.encodeBegin(FormRenderer.java:143) at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:864) at org.omnifaces.component.input.Form.encodeBegin(Form.java:135) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:302) at com.sun.faces.renderkit.html_basic.PassthroughRenderer.encodeChildren(PassthroughRenderer.java:104) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:889) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1856) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1859) at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:456) at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:133) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at org.ocpsoft.rewrite.faces.RewriteViewHandler.renderView(RewriteViewHandler.java:186) at javax.faces.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:337) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:120) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:219) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:647) at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1682) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:344) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:205) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:873) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:739) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:577) at org.apache.catalina.core.ApplicationDispatcher.doDispatch(ApplicationDispatcher.java:546) at org.apache.catalina.core.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:428) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:505) at org.apache.catalina.core.StandardHostValve.dispatchToErrorPage(StandardHostValve.java:699) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:380) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:334) at org.apache.catalina.core.StandardHostValve.postInvoke(StandardHostValve.java:232) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:735) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.URISyntaxException: Illegal character in query at index 25: /500.xhtml?q=&types=files"&sort=nameSort&order=desc at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3111) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.ocpsoft.urlbuilder.AddressBuilder.create(AddressBuilder.java:79) ... 61 more ]]

landreev commented 6 years ago

As to what to try next:

Note the item 5. from my itemized list - that this is happening at the very end of the process of rendering the page, once our page code has been successfully executed. And also the very first message on the stack trace:

Error Rendering View[/dataverse.xhtml] java.lang.IllegalArgumentException: [/dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc] is not a valid URL fragment. Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder]

Could it really be as simple as making a call from, say, the init() method of the page class and (re-)encoding the URL in the current request? Googling the exception and the message above seems like a good thing to try at this point. Other people must have dealt with this.

landreev commented 6 years ago

And then there is also the option of addressing this via silencing the logging. The first portion of the stacktrace junk is logged as .severe by javax.enterprise.resource.webcontainer.jsf.application. Which is pretty deep in the javax.enterprise hierarchy, so turning the logging off completely for this class is probably ok. The second part of the whammy, the noise cause by the 500.xhtml failure actually is a .warning-level message from javax.enterprise.web. This would be very broad, but may be ok to raise to SEVERE - ?

pdurbin commented 6 years ago

Commenting out the searchservice call, and completely removing search-include-fragment from the dataverse page still results in the same noisy failure.

I am so relieved to hear that it's not the crappy search code I wrote years ago that's causing all this noise in the logs and filling up the disk. Thanks for your very detailed, comment @landreev . "Kudos for using the power of the written word" goes to you at our next retrospective! 😄

matthew-a-dunlap commented 6 years ago

I have confirmed that this issue is due to prettyfaces. I removed it from our code in a branch and ran this curl:

curl 'http://localhost:8080/dataverse.xhtml;jsessionid=5abc0618cbf625c1f79439a431fd?q=&types=files"&sort=nameSort&order=desc'

It prints out the errors from our code (present in the current logs as well) stating the url can't be parsed. But it doesn't then trigger the numerous long stack traces we are seeing.

[2017-11-30T15:30:16.077-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.search.SearchServiceBean] [tid: _ThreadID=92 _ThreadName=http-listener-1(3)] [timeMillis: 1512073816077] [levelValue: 800] [[
  Search Syntax Error: Cannot parse 'dvObjectType:(files")': Lexical error at line 1, column 22.  Encountered: <EOF> after : "\")"]]

I also tried updating prettyfaces to its latest (3.4.1-final) version as it was documented to possibly solve the issue, as listed here: https://github.com/ocpsoft/rewrite/issues/224 . Did not help tho.

matthew-a-dunlap commented 6 years ago

From the options we have discussed before, I do not think it is feasible to re-encode the url with formatting that will not break prettyfaces. This is because as far as I understand the request cannot be modified in this way.

Gustavo and I discussed another option, similar to the one above, which is to extend the faces servlet and sanitize the url there. This seems to line up somewhat with the solution hinted at in the errors being logged Consider encoding relevant portions of the URL with [class org.ocpsoft.urlbuilder.util.Encoder] . Fwiw, mostly looks to be encoding the url with UTF-8. https://github.com/ocpsoft/rewrite/blob/master/addressbuilder/src/main/java/org/ocpsoft/urlbuilder/util/Encoder.java

I agree with Leonid's analysis of setting the logging itself. Its not great and has a small chance of us missing some errors, but probably will be ok.

matthew-a-dunlap commented 6 years ago

Discussing with Gustavo more, the extending of the faces servlet will not work with it being a pretty faces issue. We should pull this from 4.8.4 as the only option we are aware of is the logging fix and that is not a code change.

scolapasta commented 6 years ago

Latest updates: The issue seems to be with the URLs and prettyfaces. Prettyfaces expects the URL to be encoded and throws an exception. We're not sure if the fix should be on their side (better handle this situation) or our side (encode the URL). But we're going to try to add a filter that attempts to encode all urls coming in on our side; if a bad url is sent, it will forward to a 400 (Bad Request) page.

matthew-a-dunlap commented 6 years ago

I have created a filter that is able to catch the errors happening in prettyfaces before they "blow up". There is an added piece of complexity though in generating a request that does not blow up. The HttpServletRequest object does not allow editing of its query parameters, instead HttpServletRequestWrapper has to be used to create a subclass that overrides the getParameter method. I should be able to finish this up on Monday.

See here for some details: https://stackoverflow.com/questions/1413129/modify-request-parameter-with-servlet-filter

Note: Our plan is still to point these malformed requests to a 400 error page. I may be taking the wrong path in trying to modify the current request to get there, and should be trying to create a new one from scratch instead. If so I need to understand better understand our use/implementation of the HttpServletRequest interface.

pdurbin commented 6 years ago

Yesterday (Saturday) there was an update over at https://github.com/ocpsoft/rewrite/issues/254#issuecomment-348755884 suggesting a fix to the rewrite library that might fix our issue. We'd have to build it ourselves, though.

matthew-a-dunlap commented 6 years ago

I built and tested the 3.4.2 branch of prettyfaces, and it handles the malformed url much better. It turns it into a single line of error:

[2017-12-04T13:06:05.097-0500] [glassfish 4.1] [WARNING] [] [] [tid: _ThreadID=127 _ThreadName=http-listener-1(4)] [timeMillis: 1512410765097] [levelValue: 900] [[ Skipping outbound rewriting of invalid URL: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc]]

Even though the class name is not listed correctly in the logging, I was able to set a filter and get rid of the logging entirely:

asadmin set-log-levels org.ocpsoft.rewrite.servlet=SEVERE

If we decide to filter this we may want to filter in a few other places too. For example searchservicebean seems to be set at the info level atm and probably could be set to warning instead:

[2017-12-04T13:06:04.663-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.search.SearchServiceBean] [tid: _ThreadID=127 _ThreadName=http-listener-1(4)] [timeMillis: 1512410764663] [levelValue: 800] [[ Search Syntax Error: Cannot parse 'dvObjectType:(files")': Lexical error at line 1, column 22. Encountered: <EOF> after : "\")"]]

I am going to hold off of finishing the filtering code solution as this seems to be the correct way forward.

pdurbin commented 6 years ago

searchservicebean seems to be set at the info level atm and probably could be set to warning instead

Pull request #4303 is related.

djbrooke commented 6 years ago

The priority of this has decreased as the space issue in production has been resolved.

Moving to the backlog. We'll pick this up when we there's a new release of PrettyFaces.

matthew-a-dunlap commented 6 years ago

We heard back about PrettyFaces, and they have said that there is no time window for the next release and it may be a far while out. Their last release was August 2016, fwiw.

There are a few options going forward:

  1. Do nothing new and clean up the noisy logs on our servers as they come. Wait for the new release someday.
  2. Add logging.properties configuration that quiets down the noise. Adds some risk that we'll miss some warnings on core java classes (the few errors we'd be blocking are prettyfaces specific)
  3. Finish our filter that checks up front if a request is invalid and handle it quietly if so. Will take about 1 more day of development and will have a performance impact on requests (though this is likely minescule and fairly easy to test).
  4. Use the development version of the needed parts of the prettyfaces code. Live on the bleeding edge and maybe get hurt.

@djbrooke I wanted to make sure you are aware of the status and options, tho I too am leaning towards just leaving it as is.

djbrooke commented 6 years ago

Thanks!

  1. Do nothing new and clean up the noisy logs on our servers as they come. Wait for the new release someday.

With the cleaner, we've addressed the log size issue for dataverse.harvard.edu, so this has gone down in priority. We should be open to hearing from other installations where there may be similar scans and we should be willing to share our cleaner if it's appropriate to do so.

matthew-a-dunlap commented 6 years ago

@djbrooke Great!

Btw, there is actually a 5th option, which would be to take their latest release branch and add the minor fix for our issue to it and bundle it with our code. Probably the best option if we do want a more active approach someday.

kcondon commented 6 years ago

OK just remember we may want the cleaner on all our hosts since pretty much all are affected. Plus the cleaner needs to be written. I am actually in favor of 3: Finish our filter that checks up front if a request is invalid and handle it quietly if so. Will take about 1 more day of development and will have a performance impact on requests (though this is likely minescule and fairly easy to test).

Gustavo seemed to think there would be effectively no performance impact but needs to be tested.

djbrooke commented 6 years ago

Yeah, we should definitely revisit when they make the next release available.

@kcondon I was under the impression that we were already cleaning the logs. Is this not the case?

kcondon commented 6 years ago

I had not been told of that but apparently so. I still like option 3.

matthew-a-dunlap commented 6 years ago

I took a quick stab at trying to wrap up my filter for this. I ran into some issues and stopped trying, though the code as-is may be ok.

I tried to wipe out all the query parameters via HttpServletRequestWrapper, which is the documented way to alter query parameters in a filter (see https://stackoverflow.com/questions/1413129 ). The problem is that just wiping out the query parameters throws errors with (I think) primefaces and possibly other places too. I even tried to preserve the parameters primefaces seemed to want to hold onto, but what those were wasn't obvious and I stopped walking down that rabbit hole. The error in the log was Unable to locate renderkit instance for render-kit-id . Using HTML_BASIC instead. while the browser page / curl response just said No Debug Output Available.

In the past I had tried creating a new request from scratch but Java freaks out, and from everything I read modifying the request is the way to go.

I commented out the use of the wrapper in the branch (https://github.com/IQSS/dataverse/tree/test-filter-before-prettyfaces-4204). The code as is does stop the logs from filling but if a user enters a malformed url they are just given a blank page. Even trying to create one of these malformed urls is hard so In my opinion I think its ok to return blank. There is still option 5 documented above, to apply the prettyfaces fix to the most recent stable release source and build it.

matthew-a-dunlap commented 6 years ago

Prettyfaces has released a new version, 3.4.2, which includes the fix we has asked for in relation to this story: https://github.com/ocpsoft/rewrite/releases/tag/3.4.2.Final .

Barring production issues, I see little reason to not incorporate it into Dataverse and put this issue to rest :)

matthew-a-dunlap commented 6 years ago

After updating to the latest release with our fix, test curls cause much less noise (compare to https://github.com/IQSS/dataverse/issues/4204#issuecomment-348246070):

curl 'http://localhost:8080/dataverse.xhtml;jsessionid=5abc0618cbf625c1f79439a431fd?q=&types=files"&sort=nameSort&order=desc

[2018-03-07T13:33:57.285-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.search.SearchServiceBean] [tid: _ThreadID=29 _ThreadName=http-listener-1(1)] [timeMillis: 1520447637285] [levelValue: 800] [[
  Search Syntax Error: Cannot parse 'dvObjectType:(files")': Lexical error at line 1, column 22.  Encountered: <EOF> after : "\")"]]

[2018-03-07T13:33:57.286-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.search.SearchIncludeFragment] [tid: _ThreadID=29 _ThreadName=http-listener-1(1)] [timeMillis: 1520447637286] [levelValue: 800] [[
  Search Syntax Error: Cannot parse 'dvObjectType:(files")': Lexical error at line 1, column 22.  Encountered: <EOF> after : "\")"]]

[2018-03-07T13:33:57.330-0500] [glassfish 4.1] [INFO] [] [edu.harvard.iq.dataverse.search.SearchIncludeFragment] [tid: _ThreadID=29 _ThreadName=http-listener-1(1)] [timeMillis: 1520447637330] [levelValue: 800] [[
  Search Syntax Error: Cannot parse 'dvObjectType:(files")': Lexical error at line 1, column 22.  Encountered: <EOF> after : "\")"]]

[2018-03-07T13:33:57.424-0500] [glassfish 4.1] [WARNING] [] [] [tid: _ThreadID=29 _ThreadName=http-listener-1(1)] [timeMillis: 1520447637424] [levelValue: 900] [[
  Skipping outbound rewriting of invalid URL: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc]]

[2018-03-07T13:33:57.424-0500] [glassfish 4.1] [WARNING] [] [] [tid: _ThreadID=29 _ThreadName=http-listener-1(1)] [timeMillis: 1520447637424] [levelValue: 900] [[
  Skipping outbound rewriting of invalid URL: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc]]

[2018-03-07T13:33:57.425-0500] [glassfish 4.1] [WARNING] [] [] [tid: _ThreadID=29 _ThreadName=http-listener-1(1)] [timeMillis: 1520447637425] [levelValue: 900] [[
  Skipping outbound rewriting of invalid URL: /dataverse.xhtml?q=&types=files"&sort=nameSort&order=desc]]
matthew-a-dunlap commented 6 years ago

When I created this branch yesterday my pom was accidentally referencing the 3.4.2 dev branch I had built a while back. I cleared out my maven, updated the pom and now it is pointing to the actual repos (thanks @pdurbin ).

In terms of this upgrade, the only migration info I could find was for upgrading from the old ocpsoft repo with its different numbering scheme. We are on 2.0.12 of the new numbering scheme as of develop. My relatively brief tests at our rewritten urls did not show any issues upgrading to 3.4.2.

pdurbin commented 6 years ago

@matthew-a-dunlap cool, thanks for making pull request #4492 (and for adding "Final").

@chkal thanks for the fix at https://github.com/ocpsoft/rewrite/commit/67b26ae9eb27ef7946484d9a9db183d092f7a640 !

pdurbin commented 6 years ago

I just moved this issue to QA at https://waffle.io/IQSS/dataverse and approved the pull request via GitHub. The change is small enough that a screenshot is effective:

screen shot 2018-03-08 at 12 17 36 pm

I'm seeing "This is a major update, so we incremented from major version 2 to version 3" in release notes at https://www.ocpsoft.org/releases/rewrite-3-4-0-final-release-announcement/ but noting scary about upgrading. Nothing we need to do.