rafal-szypulka / grafana-ibm-apm

Grafana datasource plugin for IBM APM
MIT License
28 stars 15 forks source link

ITM TEPS still eating memory #7

Closed jripollv closed 6 years ago

jripollv commented 6 years ago

I've followed the installation instructions for version 0.6 but ITM TEPS is still eating memory. The server returns and 500 error after each DELETE method call. { "xhrStatus": "complete", "request": { "method": "DELETE", "url": "api/datasources/proxy/2/datasources/TMSAgent.%25IBM.STATIC021/datasets/MetricGroup.NTPROCSSR", "params": { "param_SourceToken": "Primary:IBERPER:NT", "optimize": "true", "param_Time": "20171028T220000--20171031T143915", "properties": "DPCTIME,TIMESTAMP,INSTCNAME", "condition": "INSTCNAME = '_Total'", "param_refId": 1402437313643 } }, "response": { "error": "Internal Server Error", "response": "Error 500: SRVE0295E: Error reported: 500\r\n", "message": "Internal Server Error" } }

rafal-szypulka commented 6 years ago

Do you access Grafana via nginx or directly? I noticed that on ITM 6 the HTTP DELETE method is rejected because of invalid Referer HTTP header (HTTP GET works). Check TEPS WAS SystemOut.log to make sure (path vary depending on platform, on Linux it is similar to: /opt/IBM/ITM/lx8266/iw/profiles/ITMProfile/logs/ITMServer/SystemOut.log), grep for Referer string. If you find something similar to:

00000014 rest          E com.ibm.tivoli.rest.RestRequestProcessor service ##### 
Rejecting request due to invalid Referer attribute

it means the ITM REST API rejects DELETE request because invalid Referer. The workaround (installation of nginx reverse proxy and clearing the Referer header) is described here: https://github.com/rafal-szypulka/grafana-ibm-apm#grafana-installation. Please let me know if it solved your problem and I will update my documentation (so far I saw this problem only for APM8).

jripollv commented 6 years ago

It looks like we have in ITM the same referer problem. This is part of the SystemOut.log file before applying the firefox referer workaround: [11/2/17 9:20:00:749 CET] 00000020 rest E com.ibm.tivoli.rest.RestRequestProcessor service ##### Rejecting request due to invalid Referer attribute:http://spmonreport:3000 [11/2/17 9:20:00:749 CET] 00000020 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[RestServlet]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: SRVE0295E: Error reported: 500 at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:626) at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:656) at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1236) at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1210) at com.ibm.tivoli.rest.RestRequestProcessor.handleRequest(RestRequestProcessor.java:243) at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457) at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025) at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3763) at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:975) at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:166) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1691) An this is after Firefox workaround: [11/2/17 9:59:44:834 CET] 0000001f rest E com.ibm.tivoli.rest.RestRequestProcessor service ##### The referer field does not exist in the HTTP request. The lack of a referer field could be considered a CSRF attack when the request is performing a state change.null

Although the memory used by TEPS increases around 400MB after the metrics loading it is more stable now. Now I've got another problem but I'm going to open another issue.

rafal-szypulka commented 6 years ago

IMO it looks fine, it is just a harmless warning. If you want to fix it, put "http://:15200" in the Referer value instead of "" (empty Referer).