search

rafal-szypulka / itm_exporter

ITM Exporter for Prometheus
MIT License
13 stars 2 forks source link

KFWITM454E Request failed due to offline managed system(s).'."} source="main.go:126" #1

Closed leondelange6969 closed 4 years ago

leondelange6969 commented 4 years ago

Please have a look and see if you can tell what up with this. Its a new installtion and for the most part its working...

ITM v6

DE Version CJ063007_tms630fp7

config.yaml itm_server_url: "http://jgapsr000000008:15200" itm_server_user: "sysadmin" itm_server_password: "xxxxxxxx" connection_timeout: 15 collection_timeout: 45 groups:

This produce the following http://jgapsr000000008:8000/metrics

HELP itm_scrape_duration_seconds Time ITM attribute group scrape took.

TYPE itm_scrape_duration_seconds gauge

itm_scrape_duration_seconds{group="KLZCPU"} 0.145134457 itm_scrape_duration_seconds{group="KLZDISK"} 0.065749022 itm_scrape_duration_seconds{group="KLZVM"} 0.065682694 itm_scrape_duration_seconds{group="QSG_CFSTR"} 0.185712258 itm_scrape_duration_seconds{group="QSG_QUEUES"} 0.145967956 itm_scrape_duration_seconds{group="SYSCPUUTIL"} 0.064833224

HELP itm_scrape_duration_seconds_total Time ITM scrape took.

TYPE itm_scrape_duration_seconds_total gauge

itm_scrape_duration_seconds_total 0.185715429

HELP itm_up itm_exporter successfully connected to the TEP data provider

TYPE itm_up gauge

itm_up 1

And the error is

ERRO[2388] {"msgId":"ATKRST100E","stackTrace":"com.ibm.usmi.console.navigator.model.NavException: (ATKRST100E) ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'.: nested exception is: \n\tcom.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.rest.RestURI.getExceptionResponse(RestURI.java:580)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2304)\n\tat sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)\n\tat java.lang.reflect.Method.invoke(Method.java:611)\n\tat org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:54)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:34)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.OptionsMethodHandler.handleRequest(OptionsMethodHandler.java:46)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:52)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)\n\tat com.ibm.tivoli.rest.RestRequestProcessor.handleRequest(RestRequestProcessor.java:381)\n\tat org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:908)\n\tat com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)\n\tat com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:277)\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)\n\tat com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:166)\n\tat com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)\n\tat com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)\n\tat com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1691)\nCaused by: com.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:838)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getLargeTableModelNodes(ITMNavReportCollection.java:317)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getAllNodes(ITMNavReportCollection.java:343)\n\tat com.ibm.tivoli.rest.navmodel.NavNodeFilter.getNodes(NavNodeFilter.java:152)\n\tat com.ibm.tivoli.rest.navmodel.NavCollection.getNodes(NavCollection.java:1191)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2287)\n\t... 73 more\nCaused by: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.TEPS.CTQuery.JVMRequest.createException(JVMRequest.java:426)\n\tat com.ibm.TEPS.CTQuery.JVMRequest.execute(JVMRequest.java:198)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:833)\n\t... 78 more\n","msgSeverity":"error","msgText":"ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'."} source="main.go:126"

I can even list the following which is telling me I have a good connection to the TEPS. tivuser@jgapsr000000008 PROD $ ./main listAttributes --attributeGroup=QSG_CFSTR --dataset=/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets +--------------------------+------------+ | DESCRIPTION | ATTRIBUTES | +--------------------------+------------+ | Origin Node | ORIGINNODE | | QSG Name | QSGNAME | | CF Struct Name | STRNAME | | Alter Date & Time | ALTERDTTM | | Struct Level | STRLEVEL | | Description (Deprecated) | STRDESC | | Recovery Supported | RECOVERY | | CF Struct Type | STRTYPE | | Struct Status | STRSTATUS | | Max Stor | MAXSIZE | | % Stor Used | USEDPCT | | Max Entries | MAXLSTE | | Used Entries | USEDLSTE | | % Entries Used | LSTEPCT | | Failure Date & Time | FAILDTTM | | Description | USTRDESC | | Offload Usage | OFFLDUSE | | Recording Time | WRITETIME | +--------------------------+------------+

I already have grafana and prometheus working with other products

Ps: I have installed the APM Datasource and that works as long as you dont load it to much.

What am I missing

Thanks Leon

rafal-szypulka commented 4 years ago

Can you temporarily reduce your config.yaml to a single attribute group? If you have Linux OS Agent connected I would recommend starting with a single group for Linux OS Agent like:

- name: "KLZCPU"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets"
  labels: ["CPUID", "ORIGINNODE"]
  metrics: ["BUSYCPU", "IDLECPU", "SYSCPU", "USRCPU", "WAITCPU"]
  managed_system_group: "*LINUX_SYSTEM"

Can you send me the logs and output of /metrics for this reduced config?

Can you also send me the output of:

./itm_exporter listAgentTypes --temsName=MFRAMEPROD_TEMS

I am not 100% sure, but maybe the problem is with the underscore character in the TEMS name. I saw similar issues in the past. Does apm grafana plugin work with this TEMS name?

leondelange6969 commented 4 years ago

Hallo....

The display from the command

tivuser@jgapsr000000008 PROD $ ./main listAgentTypes --temsName=MFRAMEPROD_TEMS +------------------------------------------+--------------------------------------------------------------------------------------+ | AGENT TYPE | DATASET URI | +------------------------------------------+--------------------------------------------------------------------------------------+ | JVM Monitor | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.JVM_Monitor/datasets | | CICSplex | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.CICSplex/datasets | | OMEGAMON for OPR | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC040/datasets | | OMEGAMON for VTAM | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC041/datasets | | Queue-Sharing Group | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets | | OMEGAMON for MVS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC042/datasets | | Universal Data Provider | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC087/datasets | | Universal Agent | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC088/datasets | | OMEGAVIEW | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC001/datasets | | Warehouse Proxy | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC122/datasets | | OMEGACENTER Gateway | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC048/datasets | | i5/OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC005/datasets | | MVS Sysplex | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC006/datasets | | QI Agent | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC128/datasets | | MVS System | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC007/datasets | | QI Broker | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC129/datasets | | SNMP HP-UX Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RmtHPOS_HP/datasets | | Performance Analyzer Warehouse Agent | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC170/datasets | | Agentless Linux OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RLinuxOS/datasets | | All Managed Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%26IBM.STATIC000/datasets | | IBM Tivoli Monitoring 5.x Endpoint Agent | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC175/datasets | | CICSPlex | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC011/datasets | | System Management Agent Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RSolOS_SMA/datasets | | CIM Solaris Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RSolOS_CIM/datasets | | IBM Tivoli Monitoring 5.x Endpoint | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC176/datasets | | Sun Management Center Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RSolOS_SMC/datasets | | UNIX OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC013/datasets | | SNMP AIX Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RmtAIXOS_AIX/datasets | | Linux OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets | | IMS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC014/datasets | | CP IRA Manager | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC137/datasets | | MVS DB2 | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC017/datasets | | Storage Subsystem | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC139/datasets | | Agentless AIX OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RmtAIXOS/datasets | | Agentless Solaris OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RSolOS/datasets | | Windows OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC021/datasets | | Summarization and Pruning Agent | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC066/datasets | | MQSERIES | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC022/datasets | | NCP | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC147/datasets | | VTAM | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC148/datasets | | Generic Configuration | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC104/datasets | | TCP/IP | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC149/datasets | | UNIX Logs | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC070/datasets | | SNMP Windows Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RemoteWinOS_SNMP/datasets | | Mainframe Networks | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC150/datasets | | CICS COMMON | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent./datasets | | OMEGAMON for IMS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC031/datasets | | OMEGAMON for DB2 | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC032/datasets | | Tivoli Enterprise Portal Server | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC153/datasets | | WebSphere Message Broker | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC154/datasets | | OMEGAMON for CICS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC033/datasets | | Agentless HP-UX OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RmtHPOS/datasets | | CICSTG | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC115/datasets | | OMEGAMON for SMS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC038/datasets | | UISolution.manager | /providers/itm.MFRAMEPROD_TEMS/datasources/UISolution.manager/datasets | | OMEGAMON for VM | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC039/datasets | | Mainframe Networks Sysplex | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC119/datasets | | CICSplex Manager | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.CICSplexes/datasets | | WMI Windows Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RemoteWinOS_WMI/datasets | | SNMP Linux Systems | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RLinuxOS_LNX/datasets | | Agentless Windows OS | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.RemoteWinOS/datasets | | CICSTG IRA Manager | /providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC081/datasets | +------------------------------------------+--------------------------------------------------------------------------------------+

leondelange6969 commented 4 years ago

I've also added just the one query...

itm_server_url: "http://jgapsr000000008:15200" itm_server_user: "sysadmin" itm_server_password: "xxxxxxxx" connection_timeout: 8 collection_timeout: 40 groups:

OUTPUT " ERRO[0016] {"msgId":"ATKRST100E","stackTrace":"com.ibm.usmi.console.navigator.model.NavException: (ATKRST100E) ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'.: nested exception is: \n\tcom.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.rest.RestURI.getExceptionResponse(RestURI.java:580)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2304)\n\tat sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)\n\tat java.lang.reflect.Method.invoke(Method.java:611)\n\tat org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:54)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:34)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.OptionsMethodHandler.handleRequest(OptionsMethodHandler.java:46)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:52)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)\n\tat com.ibm.tivoli.rest.RestRequestProcessor.handleRequest(RestRequestProcessor.java:381)\n\tat org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:908)\n\tat com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)\n\tat com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)\n\tat com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:166)\n\tat com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)\n\tat com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)\n\tat com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)\n\tat com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)\n\tat com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)\n\tat com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)\n\tat com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1691)\nCaused by: com.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:838)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getLargeTableModelNodes(ITMNavReportCollection.java:317)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getAllNodes(ITMNavReportCollection.java:343)\n\tat com.ibm.tivoli.rest.navmodel.NavNodeFilter.getNodes(NavNodeFilter.java:152)\n\tat com.ibm.tivoli.rest.navmodel.NavCollection.getNodes(NavCollection.java:1191)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2287)\n\t... 75 more\nCaused by: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.TEPS.CTQuery.JVMRequest.createException(JVMRequest.java:426)\n\tat com.ibm.TEPS.CTQuery.JVMRequest.execute(JVMRequest.java:198)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:833)\n\t... 80 more\n","msgSeverity":"error","msgText":"ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'."} source="main.go:126"

leondelange6969 commented 4 years ago

http://jgapsr000000008:8000/metrics

HELP itm_scrape_duration_seconds Time ITM attribute group scrape took.

TYPE itm_scrape_duration_seconds gauge

itm_scrape_duration_seconds{group="KLZCPU"} 0.043439354

HELP itm_scrape_duration_seconds_total Time ITM scrape took.

TYPE itm_scrape_duration_seconds_total gauge

itm_scrape_duration_seconds_total 0.043449036

HELP itm_up itm_exporter successfully connected to the TEP data provider

TYPE itm_up gauge

itm_up 1

rafal-szypulka commented 4 years ago

Could you run this command (it will do similar API request as an exporter):

curl -u sysadmin:xxxxxx "http://jgapsr000000008:15200":15200/ibm/tivoli/rest/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets/MetricGroup.KLZCPU/items?properties=BUSYCPU&param_SourceToken=*LINUX_SYSTEM"

It should produce a json with BUSYCPU values for all Linux OS agents in LINUX_SYSTEM managed system group. If it doesn't work, please double-check that LINUX_SYSTEM exists on your system and there are online Linux OS agents inside.

leondelange6969 commented 4 years ago

Interesting.... I still get the errors but we have also discovered that we have loads of OFFLINE systems.... So lets sort that first and then see where we are after that.... I'll come back to you once this is sorted. Should not be to long

tivuser@jgapsr000000008 PROD $ curl -u sysadmin:sysadmin "http://jgapsr000000008:15200/ibm/tivoli/rest/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC134/datasets/MetricGroup.KLZCPU/items?properties=BUSYCPU&param_SourceToken=*LINUX_SYSTEM" {"msgId":"ATKRST100E","stackTrace":"com.ibm.usmi.console.navigator.model.NavException: (ATKRST100E) ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'.: nested exception is: \n\tcom.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.rest.RestURI.getExceptionResponse(RestURI.java:580)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2304)\n\tat sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)\n\tat java.lang.reflect.Method.invoke(Method.java:611)\n\tat org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:54)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:34)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)\n\tat org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.OptionsMethodHandler.handleRequest(OptionsMethodHandler.java:46)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)\n\tat org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)\n\tat org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:52)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)\n\tat org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)\n\tat com.ibm.tivoli.rest.RestRequestProcessor.handleRequest(RestRequestProcessor.java:381)\n\tat org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:668)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:908)\n\tat com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)\n\tat com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)\n\tat com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:277)\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)\n\tat com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)\n\tat com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:166)\n\tat com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)\n\tat com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)\n\tat com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1691)\nCaused by: com.ibm.tivoli.monitoring.provider.navmodel.ITMRuntimeException: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:838)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getLargeTableModelNodes(ITMNavReportCollection.java:317)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getAllNodes(ITMNavReportCollection.java:343)\n\tat com.ibm.tivoli.rest.navmodel.NavNodeFilter.getNodes(NavNodeFilter.java:152)\n\tat com.ibm.tivoli.rest.navmodel.NavCollection.getNodes(NavCollection.java:1191)\n\tat com.ibm.tivoli.rest.RestProvidersURI.getDatasourceDatasetItems(RestProvidersURI.java:2287)\n\t... 73 more\nCaused by: KFWITM454E Request failed due to offline managed system(s).\n\tat com.ibm.TEPS.CTQuery.JVMRequest.createException(JVMRequest.java:426)\n\tat com.ibm.TEPS.CTQuery.JVMRequest.execute(JVMRequest.java:198)\n\tat com.ibm.tivoli.monitoring.provider.navmodel.ITMNavReportCollection.getResult(ITMNavReportCollection.java:833)\n\t... 78 more\n","msgSeverity":"error","msgText":"ATKRST100E An unexpected error occured. The error message is as follows: 'KFWITM454E Request failed due to offline managed system(s).'."}

leondelange6969 commented 4 years ago

We managed to get all the offline systems resolved....

I have a question.... Where do I get this parm value from ?

managed_system_group - the name of the managed system group, grouping agents in scope of the collection.

Is this something in the TEPS ? or do I make my own to group things....

Ive also listed the URI and this is the content.... Looks like something maybe missing here...

{"datasetsUri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets","label":"Queue-Sharing Group","version":"07.30.02.00","type":"TMSAgent","id":"TMSAgent.%IBM.STATIC163","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163","description":"KMQ:Queue-Sharing Group","providerId":"itm.MFRAMEPROD_TEMS","widgetsUri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/widgets"}

rafal-szypulka commented 4 years ago

managed_system_group - yes this is something you define/customize in TEPS: https://www.ibm.com/support/knowledgecenter/SSTFXA_6.3.0/com.ibm.itm.doc_6.3/adminuse/group_mngsysintro_tep.htm or via CLI: https://www.ibm.com/support/knowledgecenter/SSTFXA_6.3.0/com.ibm.itm.doc_6.3/cmdref/listsystemlist.htm For every agent type there should be a default managed system group like *LINUX_SYSTEM for Linux OS Agent.

leondelange6969 commented 4 years ago

Brilliant.... That's fixed the errors I've been seen... Now no more errors...

So I have this....

groups:

Which just gives this...

HELP itm_scrape_duration_seconds Time ITM attribute group scrape took.

TYPE itm_scrape_duration_seconds gauge

itm_scrape_duration_seconds{group="QSG_QUEUES"} 0.139717216

HELP itm_scrape_duration_seconds_total Time ITM scrape took.

TYPE itm_scrape_duration_seconds_total gauge

itm_scrape_duration_seconds_total 0.139740808

HELP itm_up itm_exporter successfully connected to the TEP data provider

TYPE itm_up gauge

itm_up 1

So no metrics produced....

BUT if I do the same with the APM Datasource then it works... Like this

Object xhrStatus:"complete" request:Object method:"DELETE" url:"api/datasources/proxy/19/datasources/TMSAgent.%25IBM.STATIC163/datasets/MetricGroup.QSG_QUEUES" params:Object param_SourceToken:"MQTR::MQQSG" optimize:"true" properties:"QMNAME,QNAME,CURDEPTH,TIMESTAMP,QSGNAME" condition:"CURDEPTH > 0" param_refId:1108597443510 param_Refresh:1 param_Limit:20000 response:Object

Where am I going wrong.... I just need one to work then I go from there :-)

rafal-szypulka commented 4 years ago

Unfortunately, I don't have access to the environment with this particular agent type, so I can't recreate it. That's why I asked to try first with Linux OS agent. The example config.yaml includes working examples. Another working exapmle I have is the one for zVM:

- name: "VMWORK"
  datasets_uri: '/providers/itm.xxxxxxxxxxxxx/datasources/TMSAgent.%25IBM.STATIC101/datasets'
  labels: ["ORIGINNODE","SYSID","LPARNAME","WORKLOAD",]
  metrics: ["CPPCT", "CPSECS", "CPUPCT", "CPUSECS", "SESSTIME", "VIRTPCT", "VIRTSECS", "PAGERATW", "PAGREADS", "PAGWRTS", "RESPAGES", "STORSIZE", "WKLXATT", "WKLXMOV", "WKLXPAG", "WSS", "RESPAG2G", "WKVCPU", "CPSPCT", "CPUSPCT", "VIRTSPCT", "WKLDDSOW", "WKLDPGST", "WKLDB2G", "WKLDXRD", "WKLDXWR", "WKLDXMG", "WKLDRVPG", "WKLDPGLK", "WKLDPGLG", "WKLDDASD", "WKLDSHR", "WKLDSHRM", "WKLDTVRT", "WKLDSIO", "WKLDDAIO", "WKLDAVIO", "WKLDDI98", "WKLDURIO", "WKLDACTV", "WKLDSPGS", "WKLDSPGR", "WKLDMDCI", "PAGERATW64", "PAGREADS64", "PAGWRTS64", "RESPAG2G64", "STORSIZE64", "WKLXATT64", "WSS64"]
  managed_system_group: "*OMXE_VM"

What is definitely wrong with your config:

leondelange6969 commented 4 years ago

Thanks for the quick response..... Here is the output as requested.... I snipped the output slightly as its too big for the window. I deleted a chunk out the middle but it should be good as there are full properties records.

curl -u sysadmin:xxxxxx "http://jgapsr000000008:15200/ibm/tivoli/rest/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets/MetricGroup.QSG_QUEUES/items?properties=QMNAME,QNAME,CURDEPTH,ORIGINNODE&param_SourceToken=*MQ_QSG"

{"items":[{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.ASTI.IOTF.INTERFACE.SND","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.ASTI.IOTF.INTERFACE.SND"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_0","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_0"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.LEON.MSGS02","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.LEON.MSGS02"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_1","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_1"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.LEON.MSGS03","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.LEON.MSGS03"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_2","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_2"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.ECASA.MAINFRAME.INTERFACE.RCV","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.ECASA.MAINFRAME.INTERFACE.RCV"},{"valueState":"ok","value":453,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"453"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_3","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_3"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.ASTI.IOTF.ENDORSEMENT.SND","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.ASTI.IOTF.ENDORSEMENT.SND"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_4","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_4"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"SYSTEM.QSG.UR.RESOLUTION.QUEUE","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"SYSTEM.QSG.UR.RESOLUTION.QUEUE"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_5","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_5"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.RBA.MAINFRAME.ECASA.RCV","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.RBA.MAINFRAME.ECASA.RCV"},{"valueState":"ok","value":1796,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"1796"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_6","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_6"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"SYSTEM.QSG.CHANNEL.SYNCQ","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"SYSTEM.QSG.CHANNEL.SYNCQ"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_7","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_7"},{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.DHA.MAINFRAME.INTERFACE.RCV","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.DHA.MAINFRAME.INTERFACE.RCV"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_8","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_8"},{"valueState":"ok","value":1,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"1"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_715","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_715"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"S11DOUN1AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"S11DOUN1AQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_716","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_716"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"EMDS.BACKOUT.EMAIL.UAT","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"EMDS.BACKOUT.EMAIL.UAT"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_717","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_717"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"MQSR29X01","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"MQSR29X01"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_718","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_718"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"PPCLSSIT00AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"PPCLSSIT00AQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_719","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_719"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"EMDS.BULK.EMAIL.UAT","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"EMDS.BULK.EMAIL.UAT"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_720","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_720"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"DOCQ.MLO.TEST","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"DOCQ.MLO.TEST"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_721","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_721"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"FALCONROAPP01","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"FALCONROAPP01"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_722","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_722"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"EMDS.ONLINE.SMS.DEV","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"EMDS.ONLINE.SMS.DEV"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_723","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_723"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"CMPTADEVAQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"CMPTADEVAQMGR"},{"valueState":"ok","value":4,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"4"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_724","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_724"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"MLO.HUB.DM9.RCV.UAT","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"MLO.HUB.DM9.RCV.UAT"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_725","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_725"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"IB.AOL.SUREPHRASE.DEV","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"IB.AOL.SUREPHRASE.DEV"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_726","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_726"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"TEMPIN","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"TEMPIN"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_727","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_727"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"GPPMAI01AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"GPPMAI01AQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_728","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_728"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"EMDS.BULK.SMS.MAI","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"EMDS.BULK.SMS.MAI"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_729","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_729"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"CBP.CBPX.OTMA.RCV.UAT","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"CBP.CBPX.OTMA.RCV.UAT"},{"valueState":"ok","value":77,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"77"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_730","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_730"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"MDM.CIF_AUTH.3CDP.ONLINE.UAT.RCV","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"MDM.CIF_AUTH.3CDP.ONLINE.UAT.RCV"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_731","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_731"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"BNEPP01","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"BNEPP01"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_732","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_732"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"MUCYBUATAQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"MUCYBUATAQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_733","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_733"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"ZP078I01AQMGRB","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"ZP078I01AQMGRB"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_734","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_734"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"S11DOTN1AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"S11DOTN1AQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_735","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_735"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"MDM.CASA.3CDP.CLIENTINFO.RCV.UAT","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"MDM.CASA.3CDP.CLIENTINFO.RCV.UAT"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_736","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_736"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"BEMPP02","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"BEMPP02"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_737","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_737"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"SYSTEM.IP13.XSLT.INPUT","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"SYSTEM.IP13.XSLT.INPUT"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_738","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_738"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"SAADEV01AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"SAADEV01AQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_739","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_739"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"SAADEV00AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"SAADEV00AQMGR"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_740","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_740"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"AUTO.MSG.QUEUE","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"AUTO.MSG.QUEUE"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_741","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_741"},{"properties":[{"valueState":"ok","value":"MQTR","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQTR"},{"valueState":"ok","value":"PPCLSUAT00AQMGR","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"PPCLSUAT00AQMGR"},{"valueState":"ok","value":7363,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"7363"},{"valueState":"ok","value":"MQTR::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQTR::MQQSG"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_742","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_742"}],"filteredRows":743,"identifier":"id","totalRows":743,"numRows":743}

leondelange6969 commented 4 years ago

Also when I update to this

I get loads of this

An error has occurred while serving metrics:

742 error(s) occurred:

rafal-szypulka commented 4 years ago

hmm.. I think we go in the right direction, but again I can only guess. Pls try:

- name: "QSG_QUEUES"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets"
  label: ["QMNAME", "QNAME", "QSGNAME"]
  metrics: ["CURDEPTH"]
  managed_system_group: "*MQ_QSG"

I added QSGNAME attribute as a label. Thanks for your patience

leondelange6969 commented 4 years ago

Same issue I think...

An error has occurred while serving metrics:

742 error(s) occurred:

leondelange6969 commented 4 years ago

root@jgapsr000000008 PROD # ./main listAttributeGroups --dataset=/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets +---------------------------------------------+-----------------+ | DESCRIPTION | ATTRIBUTE GROUP | +---------------------------------------------+-----------------+ | QSG_Coupling_Facility_Structure_Backups | QSG_CFBKUP | | QSG_QMgrs | QSG_QMGR | | Managed System Information | msys | | MetricGroup.OPLOG | OPLOG | | QSG_CF_Structure_SMDS_Status | QSG_CFSMDS | | QSG_Coupling_Facility_Structure_Connections | QSG_CFCONN | | QSG_Channels | QSG_CHANS | | QSG_CF_Structure_SMDS_Connection_Status | QSG_SMDSCN | | QSG_Coupling_Facility_Structures | QSG_CFSTR | | Situation Advice | advice | | Managed System Groups | mgrp | | Situation Event Information | events | | QSG_Queues | QSG_QUEUES | +---------------------------------------------+-----------------+

leondelange6969 commented 4 years ago

tivuser@jgapsr000000008 PROD $ ./main listAttributes --attributeGroup=QSG_QUEUES --dataset=/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets +----------------+------------+ | DESCRIPTION | ATTRIBUTES | +----------------+------------+ | Origin Node | ORIGINNODE | | QSG Name | QSGNAME | | Queue Name | QNAME | | CF Struct Name | STRNAME | | QSG Disp | QSGDISP | | Current Depth | CURDEPTH | | % Full | PCTFULL | | QMgr Name | QMNAME | | Recording Time | WRITETIME | +----------------+------------+

rafal-szypulka commented 4 years ago

It looks that the combination of label values and metric value has to be unique (that's the Prometheus data model) and in our case, we have duplicates for a combination of values of "QMNAME", "QNAME", "QSGNAME" and CURDEPTH. Maybe if we add ORIGINNODE and STRNAME we will receive unique sets of labels? Try different combinations of string metrics as label values. I assume that in TEP, none of the attributes we selected as labels are empty. If they are empty, please remove it from labels. What we need are only unique combinations of label values and metric value. For example:

- name: "QSG_QUEUES"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets"
  label: ["QMNAME", "QNAME", "QSGNAME", "ORIGINNODE", "STRNAME", "QSGDISP"]
  metrics: ["CURDEPTH"]
  managed_system_group: "*MQ_QSG"
leondelange6969 commented 4 years ago

Oh I see this is the issues then…. I’m not expecting just a single line back. The list has about 750+ entries on it… It’s a table with MQ Queue name and the queue depths associated with this.

Is this an issue for the plugin ? That would explain why it’s not working then.

For example:

This is one record for the queue HUB.ASTI.IOTF.INTERFACE.SND and depth = 0

{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.ASTI.IOTF.INTERFACE.SND","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.ASTI.IOTF.INTERFACE.SND"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"},{"valueState":"ok","value":"MSGS01","label":"CF Struct Name","valueType":"string","id":"STRNAME","displayValue":"MSGS01"},{"valueState":"ok","value":2,"label":"QSG Disp","valueType":"int","id":"QSGDISP","displayValue":"Shared"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_0","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_0"},

This is record two for the queue HUB.LEON.MSGS02 and depth = 0

{"properties":[{"valueState":"ok","value":"MQQP","label":"QMgr Name","valueType":"string","id":"QMNAME","displayValue":"MQQP"},{"valueState":"ok","value":"HUB.LEON.MSGS02","label":"Queue Name","valueType":"string","id":"QNAME","displayValue":"HUB.LEON.MSGS02"},{"valueState":"ok","value":0,"label":"Current Depth","valueType":"int","id":"CURDEPTH","displayValue":"0"},{"valueState":"ok","value":"MQQP::MQQSG","label":"Origin Node","valueType":"string","id":"ORIGINNODE","displayValue":"MQQP::MQQSG"},{"valueState":"ok","value":"MSGS02","label":"CF Struct Name","valueType":"string","id":"STRNAME","displayValue":"MSGS02"},{"valueState":"ok","value":2,"label":"QSG Disp","valueType":"int","id":"QSGDISP","displayValue":"Shared"}],"tooltip":"","label":"","typeLabel":"","type":"","id":"0_1","description":"","uri":"\/providers\/itm.MFRAMEPROD_TEMS\/datasources\/TMSAgent.%25IBM.STATIC163\/datasets\/MetricGroup.QSG_QUEUES\/items\/0_1"}

And so on…. There is about 750 but that should be irrelevant I guess.

Could the record be transformed to Prometheus format as its stands just now ?

rafal-szypulka commented 4 years ago

Can you confirm that the latest config I sent i.e:

- name: "QSG_QUEUES"
  datasets_uri: "/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets"
  label: ["QMNAME", "QNAME", "QSGNAME", "ORIGINNODE", "STRNAME", "QSGDISP"]
  metrics: ["CURDEPTH"]
  managed_system_group: "*MQ_QSG"

Also returns: collected metric "qsg_queues_curdepth" <....> was collected before with the same name and label values? If yes, can you send me the full output of 

curl -u sysadmin:xxxxxx "http://jgapsr000000008:15200/ibm/tivoli/rest/providers/itm.MFRAMEPROD_TEMS/datasources/TMSAgent.%25IBM.STATIC163/datasets/MetricGroup.QSG_QUEUES/items?properties=all&param_SourceToken=*MQ_QSG"

If its too long, please send it to my email rafal.szypulka@pl.ibm.com. I need to find out which combination of labels doesn't return duplicates.

leondelange6969 commented 4 years ago

Email sent this morning

leondelange6969 commented 4 years ago

HELP itm_scrape_duration_seconds Time ITM attribute group scrape took.

TYPE itm_scrape_duration_seconds gauge

itm_scrape_duration_seconds{group="QSG_QUEUES"} 0.149280197

HELP itm_scrape_duration_seconds_total Time ITM scrape took.

TYPE itm_scrape_duration_seconds_total gauge

itm_scrape_duration_seconds_total 0.149291474

HELP itm_up itm_exporter successfully connected to the TEP data provider

TYPE itm_up gauge

itm_up 1

rafal-szypulka commented 4 years ago

solved with version 0.3

leondelange6969 commented 4 years ago

Did you update the code ?

Can you forward me a new copy please…. Saving me from having to compile again..

From: Rafal Szypulka notifications@github.com Sent: 06 May 2020 14:53 To: rafal-szypulka/itm_exporter itm_exporter@noreply.github.com Cc: leondelange6969 leon.delange@dlnet.co.uk; Author author@noreply.github.com Subject: Re: [rafal-szypulka/itm_exporter] KFWITM454E Request failed due to offline managed system(s).'."} source="main.go:126" (#1)

Closed #1 https://github.com/rafal-szypulka/itm_exporter/issues/1 .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rafal-szypulka/itm_exporter/issues/1#event-3308346864 , or unsubscribe https://github.com/notifications/unsubscribe-auth/APD5NCFV7R4W2A5DPXD236DRQFMSNANCNFSM4MFB5SHA . https://github.com/notifications/beacon/APD5NCF2YOWP4M7HVLKKUHDRQFMSNA5CNFSM4MFB5SHKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOYUYV34A.gif

rafal-szypulka commented 4 years ago

The latest binary version is always here: https://github.com/rafal-szypulka/itm_exporter/releases