search

geosolutions-it / MapStore2

The solution to create and share maps, dashboards, geostories with 3D support on the web. And it is open-source too!
https://mapstore.geosolutionsgroup.com/
Other
513 stars 401 forks source link

Problems with proxy on certain servers #8685

Open offtherailz opened 2 years ago

offtherailz commented 2 years ago

Description

When proxing certain servers, the proxy fails, with empty response. This happens only in certain installations. Actually we tested with:

How to reproduce

Expected Result The layers should be visible

Current Result The layers have errors

Browser info (use this site: https://www.whatsmybrowser.org/ for non expert users) | Browser Affected | Version | |---|---| [map.zip](https://github.com/geosolutions-it/MapStore2/files/9766633/map.zip) |Internet Explorer| | |Edge| | |Chrome| | |Firefox| | |Safari| |

Other useful information

Logged error

Error in log: 

[http-nio-8080-exec-6] ERROR it.geosolutions.httpproxy.HTTPProxy - Error executing HTTP method
java.net.SocketException: Connection reset
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
    at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
    at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at it.geosolutions.httpproxy.HTTPProxy.executeProxyRequest(HTTPProxy.java:704)
    at it.geosolutions.httpproxy.HTTPProxy.doGet(HTTPProxy.java:343)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:670)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:829)

In the initial investigation we noticed that tomcat update solved the problem, but this was valid only for tomcat 8. Updating to latest tomcat 9 didn't solved anything.

Details:

Moving the war in a newer tomcat (8.5.82) solved the issue.

Replicate on MapStore binary 2022.01.02 (tomcat 8.5.69) and on our installations server.

image

Instead updating to apache-tomcat-9.0.68 didn't resolved the issue.

tdipisa commented 2 years ago

@randomorder investigating a bit more with @offtherailz it seems we are having a problem with the tomcat version. Is it possible to update it in DEV as indicated in description for a final test in DEV?

randomorder commented 2 years ago

would not be a problem but in dev we have tomcat 9.0.65. Can you confirm you have the same problem in DEV?

tdipisa commented 2 years ago

@randomorder

would not be a problem but in dev we have tomcat 9.0.65. Can you confirm you have the same problem in DEV?

yes it is not effectively possible to add a catalog source https://sdigeo-free.austrocontrol.at/geoserver/free/wms in DEV (http_proxy 1.3.0) while it is possible in c125 DEV instance where we are using http_proxy 1.1.1.

@offtherailz, it can also be a problems of the http_proxy version? Can we maybe to a try using in MS DEV the v1.1.1?

offtherailz commented 2 years ago

I re-cecked. The problems seems to happen also on binary of MapStore 2022.02.01 (I can see the same behavior, but in this case the error is not logged). Connection reset can happen for many reasons, I think it needs a debug.

tdipisa commented 2 years ago

@taba90 it would be good if you can do a quick debug to see what is happening. Then we will schedule what need to be done accordingly.

@offtherailz it would e good anyway to provide in a new comment all the details related to checks you have made so far by reporting http_proxy and tomcat versions used, where it works and where it doesn't. This is important to let @taba90 to better understand the problem and better focus it for an effective debug session.

taba90 commented 2 years ago

@tdipisa I've done a quick check and according to the documentation found "java.net.SocketException: Connection reset " happens if the server terminated the connection or the connection is lost due to network issues, so usually seems that is an exception that if raised on the client side is because the socket was closed by the server before the client could finish to read. There might be however other reasons that might involve the client side as well. It would be anyway usefull to enable logging of the httpclient library used by the proxy in order to have a deeper look. I can check this better monday

tdipisa commented 2 years ago

@taba90 ok, thank you so much. Keep me updated then.

taba90 commented 2 years ago

I need to configure debug logging level for the apache proxy library used by the http proxy and check the full logs then. @randomorder can I have access to one of the machines with the failing instances in order to set the logging configuration and restart the proxy?

tdipisa commented 2 years ago

@randomorder can you please provide what requested above?

tdipisa commented 2 years ago

@taba90 after the austrocontrol project update we experienced the same problem. Switching back to the http_proxy 1.1.1 solved the problem.

@offtherailz can you please assist @taba90 for the above?

tdipisa commented 2 years ago

Just discovered a different strange behavior here where are we using v1.3.0. Connecting to:

https://sdigeo-free.austrocontrol.at/geoserver/free/wms

I got the following error (HTTP 500):

<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> Servlet.init() for servlet [HttpProxy] threw exception</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>javax.servlet.ServletException: Servlet.init() for servlet [HttpProxy] threw exception
    org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
    org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
    org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    java.base&#47;java.lang.Thread.run(Thread.java:829)
</pre><p><b>Root Cause</b></p><pre>java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.impl.conn.ManagedHttpClientConnectionFactory
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager$InternalConnectionFactory.&lt;init&gt;(PoolingHttpClientConnectionManager.java:620)
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:181)
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:165)
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:156)
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:132)
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:123)
    it.geosolutions.httpproxy.HTTPProxy.init(HTTPProxy.java:135)
    org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
    org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
    org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    java.base&#47;java.lang.Thread.run(Thread.java:829)
</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class="line" /><h3>Apache Tomcat/9.0.65</h3></body></html>

No problem connecting to our GS stable.

randomorder commented 2 years ago

still need devops here @tdipisa ?

tdipisa commented 2 years ago

not for now @randomorder, still waiting @taba90 to look at this. See above.

taba90 commented 2 years ago

talked with @offtherailz, working on investigating this currently.

taba90 commented 2 years ago

@tdipisa I did not come to anything. Steps taken so far:

  1. Comparing the raw httprequests using Fiddler of failing proxy, successful proxy and chrome and they all seems the same.
  2. debugging source code of the http-client library to see where exactly the failures is: when reading the headers from geoserver. To proceed with the investigation I would need to check the log on geoserver side after the request fails. Currently I'm blocked on this.
tdipisa commented 2 years ago

@taba90 let's move this in blocked status for now since we don't have the GS logs from https://sdigeo-free.austrocontrol.at/geoserver/free/wms

tdipisa commented 2 years ago

@taba90 I've this error below here where 1.4-SNAPSHOT is used

<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> Servlet.init() for servlet [HttpProxy] threw exception</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>javax.servlet.ServletException: Servlet.init() for servlet [HttpProxy] threw exception
    org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
    org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:526)
    org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1651)
    org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    java.lang.Thread.run(Thread.java:748)
</pre><p><b>Root Cause</b></p><pre>java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.getDefaultRegistry(PoolingHttpClientConnectionManager.java:116)
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.&lt;init&gt;(PoolingHttpClientConnectionManager.java:123)
    it.geosolutions.httpproxy.HTTPProxy.init(HTTPProxy.java:135)
    org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:544)
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:698)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
    org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:526)
    org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1651)
    org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    java.lang.Thread.run(Thread.java:748)
</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class="line" /><h3>Apache Tomcat/8.5.69</h3></body></html>

Nothing relevant reported by the GS log with VERBOSE level. If you want we can schedule a call to look together.

taba90 commented 2 years ago

@tdipisa It seems some dependency issue. I will check depedencies version for http client in the C040 downstream project.

tdipisa commented 2 years ago

moving in blocked for now. A review of the overall MV dependency management is suggested for this.