search

inspectIT / inspectit-ocelot

inspectIT Ocelot - Java agent for collecting application performance, tracing and behavior data
http://www.inspectit.rocks/
Apache License 2.0
204 stars 69 forks source link

Inspectit-ocelot agent failed after initilaization #330

Closed sattishv closed 4 years ago

sattishv commented 5 years ago

Hi team,

We have enabled the java agent using the below listed options. The app started without issues but after few mins it exited . Have attached the stdout log gathered from server since i dont see any logs created by the agent.

wrapper.java.additional.51=-javaagent:/inspectIT/inspectit-ocelot/inspectit-ocelot-agent-0.3.jar wrapper.java.additional.52=-Dinspectit.publishOpenCensusToBootstrap=true

Log Snippets:

Startup:

Launching a JVM... INFO | jvm 1 | 2019/08/01 02:43:15 | WrapperManager: Initializing... INFO | jvm 1 | 2019/08/01 02:43:16 | 2019-08-01 02:43:16,094 INFO 422 --- [inspectIT] [ Thread-0] rocks.inspectit.ocelot.core.AgentImpl : Starting inspectIT Ocelot Agent... INFO | jvm 1 | 2019/08/01 02:43:16 | 2019-08-01 02:43:16,101 INFO 429 --- [inspectIT] [ Thread-0] rocks.inspectit.ocelot.core.AgentImpl : OpenCensus was loaded in bootstrap classloader INFO | jvm 1 | 2019/08/01 02:43:17 | INFO | jvm 1 | 2019/08/01 02:43:17 | . _ _ INFO | jvm 1 | 2019/08/01 02:43:17 | /\ / '_ () \ \ \ \ INFO | jvm 1 | 2019/08/01 02:43:17 | ( ( )\ | ' | '| | ' \/ ` | \ \ \ \ INFO | jvm 1 | 2019/08/01 02:43:17 | \/ _)| |)| | | | | || (| | ) ) ) ) INFO | jvm 1 | 2019/08/01 02:43:17 | ' |__| .|| ||| |\, | / / / / INFO | jvm 1 | 2019/08/01 02:43:17 | =========|_|==============|__/=//// INFO | jvm 1 | 2019/08/01 02:43:17 | :: Spring Boot :: (v1.5.15.RELEASE) INFO | jvm 1 | 2019/08/01 02:43:17 | INFO | jvm 1 | 2019/08/01 02:43:18 | 2019-08-01 02:43:17,996 INFO 2324 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : Registered Configuration Sources: INFO | jvm 1 | 2019/08/01 02:43:18 | 2019-08-01 02:43:17,998 INFO 2326 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : systemProperties INFO | jvm 1 | 2019/08/01 02:43:18 | 2019-08-01 02:43:18,007 INFO 2335 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : systemEnvironment INFO | jvm 1 | 2019/08/01 02:43:18 | 2019-08-01 02:43:18,007 INFO 2335 --- [inspectIT] [ Thread-0] r.i.o.core.config.InspectitEnvironment : inspectitDefaults

INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,815 INFO 10143 --- [inspectIT] [ Thread-0] r.i.o.c.m.system.GCMetricsRecorder : Enabling GC metrics recorder INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,836 INFO 10164 --- [inspectIT] [ Thread-0] i.o.c.m.s.AbstractPollingMetricsRecorder : Enabling ThreadMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,840 INFO 10168 --- [inspectIT] [ Thread-0] i.o.c.m.s.AbstractPollingMetricsRecorder : Enabling ClassLoaderMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,868 INFO 10196 --- [inspectIT] [ Thread-0] i.o.c.m.s.AbstractPollingMetricsRecorder : Enabling ProcessorMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,871 INFO 10199 --- [inspectIT] [ Thread-0] i.o.c.m.s.AbstractPollingMetricsRecorder : Enabling MemoryMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,873 INFO 10201 --- [inspectIT] [ Thread-0] i.o.c.m.s.AbstractPollingMetricsRecorder : Enabling DiskMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,888 INFO 10216 --- [inspectIT] [ Thread-0] o.c.c.p.f.ConfigurationDirectoriesPoller : Starting config directory polling service.. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,892 INFO 10220 --- [inspectIT] [ Thread-0] o.c.c.p.f.ConfigurationDirectoriesPoller : No directories to watch registered, terminating poll service. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,914 INFO 10242 --- [inspectIT] [ Thread-0] r.i.o.c.c.p.h.HttpConfigurationPoller : Starting HTTP configuration polling service. INFO | jvm 1 | 2019/08/01 02:43:25 | 2019-08-01 02:43:25,926 INFO 10254 --- [inspectIT] [ Thread-0] r.i.o.c.e.PrometheusExporterService : Starting Prometheus Exporter on

INFO | jvm 1 | 2019/08/01 02:43:57 | ---------- ConnectionManager JDBC Driver Name/Version: PayPal OCC Driver / v1.1 INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp: INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: Encountered an error running start main: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:137) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:537) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.raptor.payments.RaptorApplication.main(RaptorApplication.java:47) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.lang.reflect.Method.invoke(Method.java:497) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:316) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.lang.Thread.run(Thread.java:744) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: Caused by: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.initialize(TomcatEmbeddedServletContainer.java:138) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.(TomcatEmbeddedServletContainer.java:87) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getTomcatEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:554) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:179) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:164) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:134) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: ... 14 more INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'outerTransactionCalFilterProxy' defined in class path resource [com/paypal/raptor/springboot/rest/RestConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.web.servlet.FilterRegistrationBean]: Factory method 'outerTransactionCalFilterProxy' threw exception; nested exception is java.lang.NoClassDefFoundError: Could not initialize class com.ebay.aero.perfmon.core.ReporterRegistration INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1181) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1075) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:234) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:215) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:91) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.web.servlet.ServletContextInitializerBeans.(ServletContextInitializerBeans.java:79) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.getServletContextInitializerBeans(EmbeddedWebApplicationContext.java:241) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.selfInitialize(EmbeddedWebApplicationContext.java:228) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.access$000(EmbeddedWebApplicationContext.java:89) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext$1.onStartup(EmbeddedWebApplicationContext.java:213) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.boot.context.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:55) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5245) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1421) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1411) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.util.concurrent.FutureTask.run(FutureTask.java:266) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at io.grpc.Context$1.run(Context.java:595) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.lang.Thread.run(Thread.java:745) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.web.servlet.FilterRegistrationBean]: Factory method 'outerTransactionCalFilterProxy' threw exception; nested exception is java.lang.NoClassDefFoundError: Could not initialize class com.ebay.aero.perfmon.core.ReporterRegistration INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: ... 26 more INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.ebay.aero.perfmon.core.ReporterRegistration INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.platform.service.rest.logging.OuterTransactionCalFilter.setApplicationInfoProperties(OuterTransactionCalFilter.java:189) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.platform.service.rest.logging.OuterTransactionCalFilter.(OuterTransactionCalFilter.java:69) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.raptor.springboot.rest.RestConfig.outerTransactionCalFilterProxy(RestConfig.java:30) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.raptor.springboot.rest.RestConfig$$EnhancerBySpringCGLIB$$efd5cd24.CGLIB$outerTransactionCalFilterProxy$0() INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.raptor.springboot.rest.RestConfig$$EnhancerBySpringCGLIB$$efd5cd24$$FastClassBySpringCGLIB$$d0ce583.invoke() INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at com.paypal.raptor.springboot.rest.RestConfig$$EnhancerBySpringCGLIB$$efd5cd24.outerTransactionCalFilterProxy() INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at java.lang.reflect.Method.invoke(Method.java:497) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) INFO | jvm 1 | 2019/08/01 02:44:11 | WrapperStartStopApp Error: ... 27 more INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: Encountered an error running stop main: java.lang.reflect.InvocationTargetException INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: java.lang.reflect.InvocationTargetException INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at java.lang.reflect.Method.invoke(Method.java:497) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperStartStopApp.stop(WrapperStartStopApp.java:552) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperManager.privilegedStopInner(WrapperManager.java:4554) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperManager.access$2900(WrapperManager.java:125) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperManager$8.run(WrapperManager.java:3322) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at java.security.AccessController.doPrivileged(Native Method) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperManager.stop(WrapperManager.java:3319) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:389) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at java.lang.Thread.run(Thread.java:744) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: Caused by: java.lang.IllegalStateException: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@240b0a4a has not been refreshed yet INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1067) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1187) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: at com.paypal.raptor.springboot.core.shutdown.RaptorPrepareShutdown.main(RaptorPrepareShutdown.java:35) INFO | jvm 1 | 2019/08/01 02:44:12 | WrapperStartStopApp Error: ... 12 more INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,805 INFO 57133 --- [inspectIT] [ Thread-3] r.i.o.c.e.PrometheusExporterService : Stopping Prometheus Exporter INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,812 INFO 57140 --- [inspectIT] [ Thread-3] r.i.o.c.c.p.h.HttpConfigurationPoller : Stopping HTTP configuration polling service. INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,815 INFO 57143 --- [inspectIT] [ Thread-3] i.o.c.m.s.AbstractPollingMetricsRecorder : Disabling DiskMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,815 INFO 57143 --- [inspectIT] [ Thread-3] i.o.c.m.s.AbstractPollingMetricsRecorder : Disabling MemoryMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,816 INFO 57144 --- [inspectIT] [ Thread-3] i.o.c.m.s.AbstractPollingMetricsRecorder : Disabling ProcessorMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,816 INFO 57144 --- [inspectIT] [ Thread-3] i.o.c.m.s.AbstractPollingMetricsRecorder : Disabling ClassLoaderMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,816 INFO 57144 --- [inspectIT] [ Thread-3] i.o.c.m.s.AbstractPollingMetricsRecorder : Disabling ThreadMetricsRecorder. INFO | jvm 1 | 2019/08/01 02:44:12 | 2019-08-01 02:44:12,817 INFO 57145 --- [inspectIT] [ Thread-3] r.i.o.c.m.system.GCMetricsRecorder : Disabling GC metrics recorder STATUS | wrapper | 2019/08/01 02:44:13 | <-- Wrapper Stopped

Thanks Sattish.

JonasKunz commented 5 years ago

Hi sattisvh,

It seems like your application is unable to find the following class:

nested exception is java.lang.NoClassDefFoundError: Could not initialize class com.ebay.aero.perfmon.core.ReporterRegistration

Did you check that this error does not occur when starting without the Ocelot agent?

One potential reason why this issue could arise is because of our class loader instrumentation. You can disable it using the following option:

-Dinspectit.instrumentation.special.class-loader-delegation=false

It would be best if you also disable pushing of OpenCensus to the bootstrap, jsut to ensure that this does not cause any issues:

-Dinspectit.publishOpenCensusToBootstrap=false

You can also increase the logging granularity and specify a folder to place inspectIT's logs in using the following options, see the corresponding docuementation section:

-Dinspectit.logging.debug=true
-Dinspectit.logging.file.path=/path/to/some/log/dir
sattishv commented 5 years ago

Hi Jonas,

Thanks for the response. Have marked my inputs as inline.

Did you check that this error does not occur when starting without the Ocelot agent? yes with the agent we were able to start our service without issues. Moreover i was able to attach the agent to our process using .jattach.sh. It worked we could gather JVM and Self Monitoring metrics but not tracing data and System metrics. Thats the reason we tried to hooking the agent during the startup.

You can disable it using the following option:

-Dinspectit.instrumentation.special.class-loader-delegation=false

If we disable the above option can we get all metrics like system metrics, novatec service dependency graph.

Will also try disabling this parameter -Dinspectit.publishOpenCensusToBootstrap=false to see whether we can start the service without issues

Thanks Sattish.

On Thu, Aug 1, 2019 at 5:08 PM Jonas Kunz notifications@github.com wrote:

Hi sattisvh,

It seems like your application is unable to find the following class:

nested exception is java.lang.NoClassDefFoundError: Could not initialize class com.ebay.aero.perfmon.core.ReporterRegistration

Did you check that this error does not occur when starting without the Ocelot agent?

One potential reason why this issue could arise is because of our class loader instrumentation. You can disable it using the following option:

-Dinspectit.instrumentation.special.class-loader-delegation=false

It would be best if you also disable pushing of OpenCensus to the bootstrap, jsut to ensure that this does not cause any issues:

-Dinspectit.publishOpenCensusToBootstrap=false

You can also increase the logging granularity and specify a folder to place inspectIT's logs in using the following options, see the corresponding docuementation section https://inspectit.github.io/inspectit-ocelot/docs/configuration/logging-configuration :

-Dinspectit.logging.debug=true -Dinspectit.logging.file.path=/path/to/some/log/dir

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/inspectIT/inspectit-ocelot/issues/330?email_source=notifications&email_token=ABY2VGP7BFYIBSFL3GOLDTDQCLDLVA5CNFSM4IIPRGLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3KJAUY#issuecomment-517247059, or mute the thread https://github.com/notifications/unsubscribe-auth/ABY2VGN7LATCBLTWGCWH6JDQCLDLVANCNFSM4IIPRGLA .

-- with regards,

Sattish 9940080070

mariusoe commented 5 years ago

Hi @sattishv

so, I understand you correctly that when you disable the class loader isntrumentation with -Dinspectit.instrumentation.special.class-loader-delegation=false the agent works and provides you with all the metrics. Do you also get tracing, now?

Another question, do you already use OpenCensus in your application?

sattishv commented 5 years ago

Hi Marius,

Thanks I am yet to install the novatec-service-dependency-panel so i am not able to confirm whether tracing works or not for us. I am not able to search the panel in normal grafana plugins list. It looks like i need to build it from source is that right. Can i install service dependency panel like other plugins. Since I could install other plugins like inspectIT Ocelot - Prom - JVM Metrics very easily.

Today i was able to attach the agent without issues by disabling -Dinspectit.publishOpenCensusToBootstrap=false

We dont use OpenCensus in our application

This installation is imperative for us since i need to finalize the correct APM tools for our environment. if everything goes well we would need to install ocelot agents to all our 4000 vms.

On Mon, Aug 5, 2019 at 1:09 PM Marius Oehler notifications@github.com wrote:

Hi @sattishv https://github.com/sattishv

so, I understand you correctly that when you disable the class loader isntrumentation with -Dinspectit.instrumentation.special.class-loader-delegation=false the agent works and provides you with all the metrics. Do you also get tracing, now?

Another question, do you already use OpenCensus in your application?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/inspectIT/inspectit-ocelot/issues/330?email_source=notifications&email_token=ABY2VGL4RASKOBDH5X3FLXLQC7KLTA5CNFSM4IIPRGLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3Q6ZYA#issuecomment-518122720, or mute the thread https://github.com/notifications/unsubscribe-auth/ABY2VGLAGRFBNZAHM5RTXKDQC7KLTANCNFSM4IIPRGLA .

-- with regards,

Sattish 9940080070

mariusoe commented 5 years ago

Hi @sattishv

yes, the service dependency graph is not in the official plugin repository because it is still in an early stage. But there are already some released packages of it on its Github repository: https://github.com/NovatecConsulting/novatec-service-dependency-graph-panel/releases Using these released zip files, you can manually install the plugin as any other one: https://grafana.com/docs/plugins/installation/#installing-plugins-manually

Hmm, its odd that it is not working without that flag even you don't use the OC library.

What do you need by the sentence This installation is imperative for us since i need to finalize the correct APM tools for our environment.?

Nevertheless, if you have to roll-out the agent on that much VMs our configuration server, which will be available in the next release, might be interessting for you. Using it you can manage and roll-out configurations via its HTTP endpoint and don't have to manage all the config files manually.

sattishv commented 5 years ago

Hi mariusoe,

Thanks a lot for the response. We were able to install the novatec service dependency graph but the graph showed "no data to show" and even System Metrics graph showed no data. We could see data for only JVM metrics and Self Monitoring graph.

Our app was started with options

  1. -javaagent:/inspectIT/inspectit-ocelot/inspectit-ocelot-agent-0.3.jar
  2. -Dinspectit.publishOpenCensusToBootstrap=false

Am i missing any configuration settings.

Thanks Sattish.

mariusoe commented 5 years ago

Hi @sattishv

it is difficult to say what's the problem without knowing that much of the setup. Basically, if you use the agent without any adaption of the configuration you should see all the default metrics. Can you give some information on the environment (Java version, frameworks/libraries you're using, ...)?

It is also important to note that using the default configuration, the service graph will only show connections which are based on HTTP communication using the ApacheHttpClient, HttpURLConnectionor Servlet API and JDBCs requests. If you use any other framework or libraries for communication you have to configure it accordingly.

Anyway, you should be able to see the system metrics when you are also able to get the JVM metrics. Can you verify whether you see all metrics in the agent's Prometheus endpoint (by default at localhost:8888)?

Besides of this, if you need additional help e.g. in configuring the agent, Novatec is also offering consulting for inspectIT Ocelot and the whole topic.

All the best, Marius

sattishv commented 5 years ago

Hi Thanks for the response.

Sure here are few of the environment details. we use oracle and open jdk 1.8 versions. Tomcat as container which is wrapped with tanuki. JAXRS specification and resteasy implementation. Jackson , Netflix hystrix .Apache commons library etc httpclient : org.apache.httpcomponents httpclient 4.5.6 org.jboss.resteasy resteasy-client org.springframework spring-aop 4.3.22.RELEASE org.springframework spring-beans 4.3.22.RELEASE org.springframework spring-context 4.3.22.RELEASE org.springframework spring-core 4.3.22.RELEASE org.springframework spring-expression 4.3.22.RELEASE org.springframework spring-web 4.3.22.RELEASE org.springframework spring-webmvc 4.3.22.RELEASE org.springframework.boot spring-boot 1.5.19.RELEASE

Can you guide on how to configure http client for insoectIT so that we can apply the same approach on other components as well.

Yes we could see only JVM Metrics not the system side metrics. Have listed the output from metrics page. Even the agent log didnt show any errors or exceptions.

Metrics been collected:

HELP http_out_responsetime_sum sum of http/out/responsetime [ms]

TYPE http_out_responsetime_sum counter

http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/raptorservices/configurations/frameworkconfig/versions/1.0.0",http_status="200",service="InspectIT Agent",target_service="",} 21.510320999999998 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/AeroKernel/configurations/McfReporterConfigMXBean/versions/0.0.1",http_status="304",service="InspectIT Agent",target_service="",} 4872.894238999999 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/AeroKernel/configurations/McfReporterConfigMXBean/versions/0.0.1",http_status="200",service="InspectIT Agent",target_service="",} 62.363887 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/paymentreadserv/configurations/application/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 5008.812892000001 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/aerohystrix/configurations/HystrixThreadPool/versions/0.0.1",http_status="304",service="InspectIT Agent",target_service="",} 4505.326173000005 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/RemoteConfigClient/configurations/DiskCacheConfig/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 4518.327795999999 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/javadataaccess/configurations/ppdal-dcp/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 5020.873128999999 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/KeyMaker/configurations/keymaker/versions/4.0.0",http_status="304",service="InspectIT Agent",target_service="",} 5666.683365999996 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/aerohystrix/configurations/HystrixCommand/versions/0.0.1",http_status="200",service="InspectIT Agent",target_service="",} 21.42178 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/RemoteConfigClient/configurations/DiskCacheConfig/versions/1.0.0",http_status="SSLException",service="InspectIT Agent",target_service="",} 5.736097 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/KeyMaker/configurations/keymaker/versions/4.0.0",http_status="SSLException",service="InspectIT Agent",target_service="",} 35.504957 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/raptorservices/configurations/frameworkconfig/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 4520.582903999998 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/aerohystrix/configurations/HystrixCommand/versions/0.0.1",http_status="304",service="InspectIT Agent",target_service="",} 4432.756790000001 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/javadataaccess/configurations/ppdal-dcp/versions/1.0.0",http_status="200",service="InspectIT Agent",target_service="",} 25.713753 http_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/KeyMaker/configurations/keymaker/versions/4.0.0",http_status="200",service="InspectIT Agent",target_service="",} 32.071239

HELP http_out_count count of http/out/responsetime [ms]

TYPE http_out_count counter

http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/raptorservices/configurations/frameworkconfig/versions/1.0.0",http_status="200",service="InspectIT Agent",target_service="",} 1.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/AeroKernel/configurations/McfReporterConfigMXBean/versions/0.0.1",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/AeroKernel/configurations/McfReporterConfigMXBean/versions/0.0.1",http_status="200",service="InspectIT Agent",target_service="",} 1.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/paymentreadserv/configurations/application/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/aerohystrix/configurations/HystrixThreadPool/versions/0.0.1",http_status="304",service="InspectIT Agent",target_service="",} 321.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/RemoteConfigClient/configurations/DiskCacheConfig/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/javadataaccess/configurations/ppdal-dcp/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/KeyMaker/configurations/keymaker/versions/4.0.0",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/aerohystrix/configurations/HystrixCommand/versions/0.0.1",http_status="200",service="InspectIT Agent",target_service="",} 1.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/RemoteConfigClient/configurations/DiskCacheConfig/versions/1.0.0",http_status="SSLException",service="InspectIT Agent",target_service="",} 1.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/KeyMaker/configurations/keymaker/versions/4.0.0",http_status="SSLException",service="InspectIT Agent",target_service="",} 1.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/raptorservices/configurations/frameworkconfig/versions/1.0.0",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/aerohystrix/configurations/HystrixCommand/versions/0.0.1",http_status="304",service="InspectIT Agent",target_service="",} 320.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/javadataaccess/configurations/ppdal-dcp/versions/1.0.0",http_status="200",service="InspectIT Agent",target_service="",} 1.0 http_out_count{application="",host="localhost",host_address="0.0.0.0",http_host="localhost:15080",http_method="GET",http_path="/v1/appconfig/projects/KeyMaker/configurations/keymaker/versions/4.0.0",http_status="200",service="InspectIT Agent",target_service="",} 1.0

HELP service_in_count count of service/in/responsetime [ms]

TYPE service_in_count counter

service_in_count{application="",host="localhost",host_address="0.0.0.0",origin_external="",origin_service="",protocol="http",service="InspectIT Agent",} 1317.0

HELP service_in_responsetime_sum sum of service/in/responsetime [ms]

TYPE service_in_responsetime_sum counter

service_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",origin_external="",origin_service="",protocol="http",service="InspectIT Agent",} 5309.245061000002

HELP http_in_responsetime_sum sum of http/in/responsetime [ms]

TYPE http_in_responsetime_sum counter

http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin.back",http_status="403",origin_service="",service="InspectIT Agent",} 0.8140379999999999 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/header.php",http_status="403",origin_service="",service="InspectIT Agent",} 0.945121 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/properties/configuration.php",http_status="403",origin_service="",service="InspectIT Agent",} 1.7311069999999997 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/MSWSMTP/Common/Authentication/Logon.aspx",http_status="403",origin_service="",service="InspectIT Agent",} 0.989667 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/properties/description.dhtml",http_status="403",origin_service="",service="InspectIT Agent",} 0.981386 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 804.949487000001 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="",http_status="404",origin_service="",service="InspectIT Agent",} 512.210886 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/aboutprinter.html",http_status="403",origin_service="",service="InspectIT Agent",} 0.9175209999999999 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/robots.txt",http_status="403",origin_service="",service="InspectIT Agent",} 1.029475 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/wavemaster.internal",http_status="403",origin_service="",service="InspectIT Agent",} 1.610768 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin/v3console/ValidateInternals",http_status="200",origin_service="",service="InspectIT Agent",} 418.921444 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/login.html",http_status="403",origin_service="",service="InspectIT Agent",} 1.312095 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/sitemap.xml",http_status="403",origin_service="",service="InspectIT Agent",} 0.891168 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/",http_status="403",origin_service="",service="InspectIT Agent",} 80.68939200000001 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="HEAD",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 2.006215 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="PROPFIND",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 0.545648 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="HEAD",http_path="/",http_status="403",origin_service="",service="InspectIT Agent",} 1.8301859999999999 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="",http_status="200",origin_service="",service="InspectIT Agent",} 1433.0693339999993 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/error",http_status="404",origin_service="",service="InspectIT Agent",} 5.12472 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin/v3console/ViewJMXMBeans",http_status="200",origin_service="",service="InspectIT Agent",} 2021.2154389999996 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/index.html",http_status="403",origin_service="",service="InspectIT Agent",} 0.9225169999999999 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="NESSUS",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 0.8512 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="OPTIONS",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 0.912372 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="USFYQP",http_path="/",http_status="403",origin_service="",service="InspectIT Agent",} 0.681561 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/file",http_status="403",origin_service="",service="InspectIT Agent",} 0.828865 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="SEARCH",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 0.693349 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="TRACK",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 0.5015419999999999 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/.cobalt",http_status="403",origin_service="",service="InspectIT Agent",} 0.9916889999999999 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="POST",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 18.225006 http_in_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin_ui/mas/ent/login.html",http_status="403",origin_service="",service="InspectIT Agent",} 0.53813

HELP http_in_count count of http/in/responsetime [ms]

TYPE http_in_count counter

http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin.back",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/header.php",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/properties/configuration.php",http_status="403",origin_service="",service="InspectIT Agent",} 2.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/MSWSMTP/Common/Authentication/Logon.aspx",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/properties/description.dhtml",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 1106.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="",http_status="404",origin_service="",service="InspectIT Agent",} 6.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/aboutprinter.html",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/robots.txt",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/wavemaster.internal",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin/v3console/ValidateInternals",http_status="200",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/login.html",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/sitemap.xml",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/",http_status="403",origin_service="",service="InspectIT Agent",} 15.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="HEAD",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 3.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="PROPFIND",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="HEAD",http_path="/",http_status="403",origin_service="",service="InspectIT Agent",} 3.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="",http_status="200",origin_service="",service="InspectIT Agent",} 72.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/error",http_status="404",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin/v3console/ViewJMXMBeans",http_status="200",origin_service="",service="InspectIT Agent",} 70.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/index.html",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="NESSUS",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="OPTIONS",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="USFYQP",http_path="/",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/file",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="SEARCH",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="TRACK",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/.cobalt",http_status="403",origin_service="",service="InspectIT Agent",} 1.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="POST",http_path="",http_status="403",origin_service="",service="InspectIT Agent",} 19.0 http_in_count{application="",host="localhost",host_address="0.0.0.0",http_method="GET",http_path="/admin_ui/mas/ent/login.html",http_status="403",origin_service="",service="InspectIT Agent",} 1.0

HELP jvm_gc_concurrent_phase_time sum of the time spent in concurrent GC

phases [ms]

TYPE jvm_gc_concurrent_phase_time counter

jvm_gc_concurrent_phase_time{action="end of major GC",cause="No GC",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 47360.0

HELP jvm_gc_memory_allocated sum of the increase in the size of the young

generation memory pool after one GC to before the next [bytes]

TYPE jvm_gc_memory_allocated counter

jvm_gc_memory_allocated{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 9.4388433608E10

HELP jvm_gc_pause sum of the time spent in GC pause [ms]

TYPE jvm_gc_pause counter

jvm_gc_pause{action="end of minor GC",cause="Allocation Failure",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 37758.0

HELP jvm_threads_peak last value of the peak number of live threads since

the JVM start [threads]

TYPE jvm_threads_peak gauge

jvm_threads_peak{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 146.0

HELP jvm_gc_live_data_size last value of the size of the old generation

memory pool after a full GC [bytes]

TYPE jvm_gc_live_data_size gauge

jvm_gc_live_data_size{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 2.68795632E8

HELP disk_free last value of free disk space [bytes]

TYPE disk_free gauge

disk_free{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 1.780549632E10

HELP jvm_memory_used last value of the amount of used memory [bytes]

TYPE jvm_memory_used gauge

jvm_memory_used{area="heap",host="localhost",host_address="0.0.0.0",id="CMS Old Gen",service="InspectIT Agent",} 3.92346936E8 jvm_memory_used{area="heap",host="localhost",host_address="0.0.0.0",id="Par Eden Space",service="InspectIT Agent",} 2.21152752E8 jvm_memory_used{area="nonheap",host="localhost",host_address="0.0.0.0",id="Metaspace",service="InspectIT Agent",} 1.27136208E8 jvm_memory_used{area="nonheap",host="localhost",host_address="0.0.0.0",id="Code Cache",service="InspectIT Agent",} 5.9825728E7 jvm_memory_used{area="nonheap",host="localhost",host_address="0.0.0.0",id="Compressed Class Space",service="InspectIT Agent",} 1.5736432E7 jvm_memory_used{area="heap",host="localhost",host_address="0.0.0.0",id="Par Survivor Space",service="InspectIT Agent",} 8.388608E7

HELP system_cpu_count last value of the number of processors available to

the JVM [cores]

TYPE system_cpu_count gauge

system_cpu_count{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 2.0

HELP service_out_count count of service/out/responsetime [ms]

TYPE service_out_count counter

service_out_count{application="",host="localhost",host_address="0.0.0.0",protocol="http",service="InspectIT Agent",target_external="localhost:15080",target_service="",} 2568.0

HELP service_out_responsetime_sum sum of service/out/responsetime [ms]

TYPE service_out_responsetime_sum counter

service_out_responsetime_sum{application="",host="localhost",host_address="0.0.0.0",protocol="http",service="InspectIT Agent",target_external="localhost:15080",target_service="",} 38698.66417599999

HELP inspectit_self_instrumented_classes last value of the number of

classes currently instrumented by inspectIT [classes]

TYPE inspectit_self_instrumented_classes gauge

inspectit_self_instrumented_classes{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 110.0

HELP jvm_buffer_total_capacity last value of an estimate of the total

capacity of the buffers in this pool [bytes]

TYPE jvm_buffer_total_capacity gauge

jvm_buffer_total_capacity{host="localhost",host_address="0.0.0.0",id="direct",service="InspectIT Agent",} 3.4159472E7 jvm_buffer_total_capacity{host="localhost",host_address="0.0.0.0",id="mapped",service="InspectIT Agent",} 0.0

HELP jvm_buffer_memory_used last value of an estimate of the memory that

the JVM is using for this buffer pool [bytes]

TYPE jvm_buffer_memory_used gauge

jvm_buffer_memory_used{host="localhost",host_address="0.0.0.0",id="direct",service="InspectIT Agent",} 3.4159473E7 jvm_buffer_memory_used{host="localhost",host_address="0.0.0.0",id="mapped",service="InspectIT Agent",} 0.0

HELP inspectit_self_duration sum of inspectIT Ocelot self-monitoring

duration [us]

TYPE inspectit_self_duration counter

inspectit_self_duration{component_name="DiskMetricsRecorder",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 1584578.0 inspectit_self_duration{component_name="classloader-delegation",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 823586.0 inspectit_self_duration{component_name="class-discovery",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 1917409.0 inspectit_self_duration{component_name="GCMetricsRecorder",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 106105.0 inspectit_self_duration{component_name="HookManager",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 5292041.0 inspectit_self_duration{component_name="ThreadMetricsRecorder",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 1.6552748E7 inspectit_self_duration{component_name="ClassLoaderMetricsRecorder",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 256733.0 inspectit_self_duration{component_name="ProcessorMetricsRecorder",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 2612130.0 inspectit_self_duration{component_name="instrumentation-analysis",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 1.7541625E7 inspectit_self_duration{component_name="InstrumentationManager",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 3892893.0 inspectit_self_duration{component_name="MemoryMetricsRecorder",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 2866297.0 inspectit_self_duration{component_name="instrumentation-retransformation",host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 9979210.0

HELP system_load_average_1m last value of the sum of the number of

runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time [percentage]

TYPE system_load_average_1m gauge

system_load_average_1m{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 0.12

HELP jvm_classes_unloaded last value of total number of classes which

have been unloaded since the start of the JVM [classes]

TYPE jvm_classes_unloaded gauge

jvm_classes_unloaded{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 122.0

HELP process_cpu_usage last value of the recent cpu usage for the JVM's

process [cores]

TYPE process_cpu_usage gauge

process_cpu_usage{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 0.013157894736842105

HELP jvm_memory_committed last value of the amount of memory that is

committed for the JVM to use [bytes]

TYPE jvm_memory_committed gauge

jvm_memory_committed{area="heap",host="localhost",host_address="0.0.0.0",id="CMS Old Gen",service="InspectIT Agent",} 1.4680064E9 jvm_memory_committed{area="heap",host="localhost",host_address="0.0.0.0",id="Par Eden Space",service="InspectIT Agent",} 6.7108864E8 jvm_memory_committed{area="nonheap",host="localhost",host_address="0.0.0.0",id="Metaspace",service="InspectIT Agent",} 1.30449408E8 jvm_memory_committed{area="nonheap",host="localhost",host_address="0.0.0.0",id="Code Cache",service="InspectIT Agent",} 6.0424192E7 jvm_memory_committed{area="nonheap",host="localhost",host_address="0.0.0.0",id="Compressed Class Space",service="InspectIT Agent",} 1.62816E7 jvm_memory_committed{area="heap",host="localhost",host_address="0.0.0.0",id="Par Survivor Space",service="InspectIT Agent",} 8.388608E7

HELP jvm_threads_states last value of the number of live threads for each

state [threads]

TYPE jvm_threads_states gauge

jvm_threads_states{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",state="BLOCKED",} 0.0 jvm_threads_states{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",state="NEW",} 0.0 jvm_threads_states{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",state="TERMINATED",} 0.0 jvm_threads_states{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",state="RUNNABLE",} 21.0 jvm_threads_states{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",state="WAITING",} 60.0 jvm_threads_states{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",state="TIMED_WAITING",} 63.0

HELP inspectit_self_instrumentation_queue_size last value of the number

of pending classes inspectIT has to check if they require instrumentation updates [classes]

TYPE inspectit_self_instrumentation_queue_size gauge

inspectit_self_instrumentation_queue_size{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 0.0

HELP method_duration_count count of the duration from method entry to

method exit [ms]

TYPE method_duration_count counter

HELP method_duration_sum sum of the duration from method entry to method

exit [ms]

TYPE method_duration_sum counter

HELP system_cpu_usage last value of the recent cpu usage for the whole

system [cores]

TYPE system_cpu_usage gauge

system_cpu_usage{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 0.02631578947368421

HELP jvm_threads_live last value of the current number of total live

threads [threads]

TYPE jvm_threads_live gauge

jvm_threads_live{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 144.0

HELP jvm_buffer_count last value of an estimate of the number of buffers

in the pool [buffers]

TYPE jvm_buffer_count gauge

jvm_buffer_count{host="localhost",host_address="0.0.0.0",id="direct",service="InspectIT Agent",} 79.0 jvm_buffer_count{host="localhost",host_address="0.0.0.0",id="mapped",service="InspectIT Agent",} 0.0

HELP jvm_classes_loaded last value of total number of classes currently

loaded in the JVM [classes]

TYPE jvm_classes_loaded gauge

jvm_classes_loaded{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 22647.0

HELP jvm_gc_max_data_size last value of the maximum size of the old

generation memory pool [bytes]

TYPE jvm_gc_max_data_size gauge

jvm_gc_max_data_size{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 1.4680064E9

HELP jvm_gc_memory_promoted sum of the count of positive increases in the

size of the old generation memory pool before GC to after GC [bytes]

TYPE jvm_gc_memory_promoted counter

jvm_gc_memory_promoted{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 6.0387088E9

HELP disk_total last value of total disk space [bytes]

TYPE disk_total gauge

disk_total{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 6.3379075072E10

HELP jvm_memory_max last value of the maximum amount of memory that can

be used for memory management [bytes]

TYPE jvm_memory_max gauge

jvm_memory_max{area="heap",host="localhost",host_address="0.0.0.0",id="CMS Old Gen",service="InspectIT Agent",} 1.4680064E9 jvm_memory_max{area="heap",host="localhost",host_address="0.0.0.0",id="Par Eden Space",service="InspectIT Agent",} 6.7108864E8 jvm_memory_max{area="nonheap",host="localhost",host_address="0.0.0.0",id="Metaspace",service="InspectIT Agent",} 1.073741824E9 jvm_memory_max{area="nonheap",host="localhost",host_address="0.0.0.0",id="Code Cache",service="InspectIT Agent",} 2.5165824E8 jvm_memory_max{area="nonheap",host="localhost",host_address="0.0.0.0",id="Compressed Class Space",service="InspectIT Agent",} 1.073741824E9 jvm_memory_max{area="heap",host="localhost",host_address="0.0.0.0",id="Par Survivor Space",service="InspectIT Agent",} 8.388608E7

HELP jvm_threads_daemon last value of the current number of live daemon

threads [threads]

TYPE jvm_threads_daemon gauge

jvm_threads_daemon{host="localhost",host_address="0.0.0.0",service="InspectIT Agent",} 138.0

On Fri, Aug 9, 2019 at 2:28 PM Marius Oehler notifications@github.com wrote:

Hi @sattishv https://github.com/sattishv

it is difficult to say what's the problem without knowing that much of the setup. Basically, if you use the agent without any adaption of the configuration you should see all the default metrics. Can you give some information on the environment (Java version, frameworks/libraries you're using, ...)?

It is also important to note that using the default configuration, the service graph will only show connections which are based on HTTP communication using the ApacheHttpClient, HttpURLConnection or Servlet API and JDBCs requests. If you use any other framework or libraries for communication you have to configure it accordingly.

Anyway, you should be able to see the system metrics when you are also able to get the JVM metrics. Can you verify whether you see all metrics in the agent's Prometheus endpoint (by default at localhost:8888)?

Besides of this, if you need additional help e.g. in configuring the agent, Novatec is also offering consulting for inspectIT Ocelot and the whole topic.

All the best, Marius

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/inspectIT/inspectit-ocelot/issues/330?email_source=notifications&email_token=ABY2VGI6NTSFRMHMS4TOGY3QDUWVDA5CNFSM4IIPRGLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD36CLII#issuecomment-519841185, or mute the thread https://github.com/notifications/unsubscribe-auth/ABY2VGLHOF2N7PL7AYO3GJ3QDUWVDANCNFSM4IIPRGLA .

-- with regards,

Sattish 9940080070

mariusoe commented 5 years ago

Hi @sattishv

thanks for your details.

Yes we could see only JVM Metrics not the system side metrics. Have listed the output from metrics page. Even the agent log didnt show any errors or exceptions.

According to the metrics you included in your comment, inspectIT is actually collecting and providing system metrics like disk_free, system_load_average, ... So, can you check if those metrics are stored in your database? If you use the default dashboards, maybe you can check if some of the queries is not correct or is not fitting the actual metric names.

Just to be sure, are you using some component like "Jaeger Agent/Collector" to collect and transfer the metrics? In that case, there are some bugs in those componetns which can cause a metrics renaming.

Can you guide on how to configure http client for insoectIT so that we can apply the same approach on other components as well.

Sure. What you have to do is to create a new configuration fitting your needs. It is required to propagate some data with your HTTP requests which can be used to correlate the sending and receiving side. (https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/rules#data-propagation)

For this you have to create scopes which specify the methods you want to target and where you can inject the context information needed (https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/scopes). In order to inject this context into your carrier medium (HTTP request) you have to specify an action (https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/rules#actions) which contains the logic to actual do the injection. In the end, you have to conncet the scope and actions using a rule (https://inspectit.github.io/inspectit-ocelot/docs/instrumentation/rules#defining-rules).

If you need an example, you can find the existing configuration for e.g. the ApacheClient or HttpUrlConnection in the configuration project: https://github.com/inspectIT/inspectit-ocelot/tree/master/inspectit-ocelot-config/src/main/resources/rocks/inspectit/ocelot/config/default/instrumentation/propagation

I hope this is helping you!

Best, Marius

mariusoe commented 4 years ago

Hi @sattishv

I hope the problem has been solved. If you need more help, feel free to reopen this issue, because I'll close it for now.