possibly related to this update to the pidsvc.war, From an email from William Francis: One key finding from our recent deployment upgrade was the java/ postgres version dependencies were causing some timeout issues – we found these were resolved by updating the pom.xml in deployment of the pid service. See the pull request on the main stack here. Our repo is here
Errors from curl:
"Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
Reason: Error reading from remote server"
From tomcat log
WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [pidsvc] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Stack trace of request processing thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:590) org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:443) org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:364) org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1525) csiro.pidsvc.mappingstore.Manager.<init>(Manager.java:204) csiro.pidsvc.servlet.dispatcher.doGet(dispatcher.java:82) javax.servlet.http.HttpServlet.service(HttpServlet.java:622) javax.servlet.http.HttpServlet.service(HttpServlet.java:729) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80) org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502) org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1152) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684) org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2464) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Thread.java:748)
This request (when the redirectee is up) stops the dispatcher from working and freezes the GUI
curl -k -l "https://geoconnex.us/chyld-pilot/id/nat_aq/N400NYNECB?f=jsonld&callback=jQuery32103053047573287002_1583436292303&_=1583436292422",
which makes a number of simultaneous requests.
Current best guess re Eric Boisvert is something to do with connection pooling.
relevant context.xml
possibly related to this update to the pidsvc.war, From an email from William Francis: One key finding from our recent deployment upgrade was the java/ postgres version dependencies were causing some timeout issues – we found these were resolved by updating the pom.xml in deployment of the pid service. See the pull request on the main stack here. Our repo is here
Errors from curl:
From tomcat log
pg_stat_activity (need to redo on replication)