search

open-telemetry / opentelemetry-java-instrumentation

OpenTelemetry auto-instrumentation and instrumentation libraries for Java
https://opentelemetry.io
Apache License 2.0
1.96k stars 859 forks source link

Broadcom Layer7 Api-Gateway Service is not starting properly after integrating with opentelemetry #8464

Closed vkroy100 closed 1 year ago

vkroy100 commented 1 year ago

Hi team we are trying to integrate opentelemetry with our API-Gateway which is layer7 Api-Gateway of Broadcom. We are getting the below error at the time of startup of the service after including the opentelemetry java-agent. Please help us in resolving this issue. Error:

java.lang.NoClassDefFoundError: Could not initialize class io.opentelemetry.javaagent.instrumentation.internal.classloader.BootDelegationInstrumentation$Holder at com.l7tech.server.policy.module.j.loadClass(Unknown Source) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) at com.l7tech.server.policy.module.ModularAssertionsScanner.onModuleLoad(Unknown Source) at com.l7tech.server.policy.module.ModulesScanner.processNewOrChangedModules(Unknown Source) at com.l7tech.server.policy.module.ModulesScanner.scanModules(Unknown Source) at com.l7tech.server.policy.module.ModularAssertionsScanner.scanModules(Unknown Source) at com.l7tech.server.policy.ServerAssertionRegistry.b(Unknown Source) at com.l7tech.server.policy.ServerAssertionRegistry.afterPropertiesSet(Unknown Source) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.lambda$invokeInitMethods$5(AbstractAutowireCapableBeanFactory.java:1836) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1835) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1413) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:330) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:113) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:428) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:173) at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:707) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:198) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1354) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at org.springframework.beans.factory.support.AbstractBeanFactory.getTypeForFactoryBean(AbstractBeanFactory.java:1709) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:899) at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:637) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:583) at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:550) at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:265) at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1546) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1343) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1413) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at com.l7tech.server.boot.GatewayBoot.e(Unknown Source) at com.l7tech.server.boot.GatewayBoot.start(Unknown Source) at com.l7tech.server.boot.GatewayBoot.runUntilShutdown(Unknown Source) at com.l7tech.server.boot.GatewayMain.main(Unknown Source) 2023-05-08T20:15:38.965+0530 INFO 55 com.l7tech.server.EntityVersionChecker: Version checking took 130077ms 2023-05-08T20:16:59.718+0530 INFO 1 com.l7tech.server.service.ServiceMetricsServicesImpl: Enabling service metrics collection. 2023-05-08T20:17:07.479+0530 INFO 58 com.l7tech.server.service.ServiceMetricsServicesImpl: Database flusher beginning 2023-05-08T20:17:07.492+0530 INFO 1 com.l7tech.server.service.ServiceMetricsServicesImpl: Using default value (3900000) for missing system property: com.l7tech.service.metrics.maxFineAge 2023-05-08T20:17:07.492+0530 INFO 1 com.l7tech.server.service.ServiceMetricsServicesImpl: Using default value (604800000) for missing system property: com.l7tech.service.metrics.maxHourlyAge 2023-05-08T20:17:07.492+0530 INFO 1 com.l7tech.server.service.ServiceMetricsServicesImpl: Using default value (31536000000) for missing system property: com.l7tech.service.metrics.maxDailyAge 2023-05-08T20:17:13.341+0530 WARNING 1 com.l7tech.server.SimplePropertyChangeHandler: 2420: Cannot parse content-type value 'multipart/form-data' from cluster property. Reason: Content-Type of type multipart must include a boundary parameter (RFC 2045 sec 5) 2023-05-08T20:17:13.362+0530 INFO 1 com.l7tech.server.identity.ldap.LdapRuntimeConfig: Read system value maxLdapSearchResultSize of 1000 2023-05-08T20:17:13.369+0530 INFO 1 com.l7tech.server.extension.registry.sharedstate.SharedKeyValueStoreProviderRegistry: Registered extension com.l7tech.server.extension.provider.sharedstate.LocalKeyValueStoreProvider with key local 2023-05-08T20:17:31.805+0530 INFO 1 com.l7tech.server.security.cert.CertValidationProcessorImpl: (Re-)Populating certificate validation caches. 2023-05-08T20:17:49.893+0530 INFO 1 com.l7tech.server.transport.http.AnonymousSslClientSocketFactory: SSL Client TrustManager initialized. 2023-05-08T20:17:49.894+0530 INFO 1 com.l7tech.server.transport.http.AnonymousSslClientHostnameAwareSocketFactory: SSL Client TrustManager initialized. 2023-05-08T20:17:49.895+0530 INFO 1 com.l7tech.server.transport.http.AnonymousSslClientHostnameAwareSocketFactory: SSL Client HostnameVerifier initialized. 2023-05-08T20:17:49.896+0530 INFO 1 com.l7tech.server.transport.http.SslClientSocketFactory: SSL Client TrustManager initialized. 2023-05-08T20:17:49.896+0530 INFO 1 com.l7tech.server.transport.http.SslClientSocketFactory: SSL Client KeyManager initialized. 2023-05-08T20:17:49.898+0530 INFO 1 com.l7tech.server.transport.http.SslClientHostnameAwareSocketFactory: SSL Client TrustManager initialized. 2023-05-08T20:17:49.899+0530 INFO 1 com.l7tech.server.transport.http.SslClientHostnameAwareSocketFactory: SSL Client KeyManager initialized. 2023-05-08T20:17:49.899+0530 INFO 1 com.l7tech.server.transport.http.SslClientHostnameAwareSocketFactory: SSL Client HostnameVerifier initialized. 2023-05-08T20:17:49.965+0530 INFO 1 com.l7tech.server.policy.module.ModulesScanner: Checking module with updated timestamp: CustomCryptoAssertion.jar 2023-05-08T20:17:49.973+0530 SEVERE 1 com.l7tech.server.policy.module.ModulesScanner: Unable to load assertion module [CustomCryptoAssertion.jar] (ignoring it until it changes): Could not load module jar '/opt/SecureSpan/Gateway/node/default/../../runtime/modules/lib/CustomCryptoAssertion.jar'. com.l7tech.server.policy.module.ModuleException: Could not load module jar '/opt/SecureSpan/Gateway/node/default/../../runtime/modules/lib/CustomCryptoAssertion.jar'. at com.l7tech.server.policy.module.CustomAssertionsScanner.a(Unknown Source) at com.l7tech.server.policy.module.CustomAssertionsScanner.onModuleLoad(Unknown Source) at com.l7tech.server.policy.module.ModulesScanner.processNewOrChangedModules(Unknown Source) at com.l7tech.server.policy.module.ModulesScanner.scanModules(Unknown Source) at com.l7tech.server.policy.module.CustomAssertionsScanner.scanModules(Unknown Source) at com.l7tech.server.policy.custom.CustomAssertionsRegistrarImpl.afterPropertiesSet(Unknown Source) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.lambda$invokeInitMethods$5(AbstractAutowireCapableBeanFactory.java:1836) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1835) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:602) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at com.l7tech.server.boot.GatewayBoot.e(Unknown Source) at com.l7tech.server.boot.GatewayBoot.start(Unknown Source) at com.l7tech.server.boot.GatewayBoot.runUntilShutdown(Unknown Source) at com.l7tech.server.boot.GatewayMain.main(Unknown Source) Caused by: java.util.zip.ZipException: zip END header not found at java.base/java.util.zip.ZipFile$Source.zerror(ZipFile.java:1597) at java.base/java.util.zip.ZipFile$Source.findEND(ZipFile.java:1488) at java.base/java.util.zip.ZipFile$Source.initCEN(ZipFile.java:1495) at java.base/java.util.zip.ZipFile$Source.(ZipFile.java:1299) at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1262) at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:733) at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:842) at java.base/java.util.zip.ZipFile.(ZipFile.java:248) at java.base/java.util.zip.ZipFile.(ZipFile.java:177) at java.base/java.util.jar.JarFile.(JarFile.java:350) at java.base/java.util.jar.JarFile.(JarFile.java:321) at java.base/java.util.jar.JarFile.(JarFile.java:287) ... 23 more

laurit commented 1 year ago

java.lang.NoClassDefFoundError: Could not initialize class io.opentelemetry.javaagent.instrumentation.internal.classloader.BootDelegationInstrumentation$Holder Means that static initializer of BootDelegationInstrumentation$Holder has previously failed, there should be an ExceptionInInitializerError before the NoClassDefFoundError. If you look at the class in question https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/df9d1a7a2648525029788a704e1f9475f257c653/instrumentation/internal/internal-class-loader/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/internal/classloader/BootDelegationInstrumentation.java#L76 then at least I can't see an obvious reason for it to fail. Perhaps there is an exception and then there is another exception when logging the first exception? I would start from investigating what fails in that static initializer. As the Broadcom Layer7 Api-Gateway does not seem to be something that we can download and run you'll have to do the debugging yourself. Or if you can, seek assistance from the api gateway vendor or your open telemetry vendor.

github-actions[bot] commented 1 year ago

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.