search

wso2 / product-ei

An open source, a high-performance hybrid integration platform that allows developers quick integration with any application, data, or system.
https://wso2.com/integration/
Apache License 2.0
374 stars 280 forks source link

LO4J Kafka Appender #2786

Closed Jean-Marc-Henry closed 1 month ago

Jean-Marc-Henry commented 6 years ago

Description: I put a kafka appender in log4J conf and add librairies on EI/lib, it works on 6.1.1 and 6.2 but no more in 6.3 and 6.4

Suggested Labels: log4j, Kafka

**Affected Product Version:6.3 and 6.4

OS, DB, other environment details and versions:
MACOS or debian

Steps to reproduce: add the folowing lines in log4j.conf at the beginning

log4j.rootLogger=ERROR, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG, ERROR_LOGFILE, KAFKA log4j.logger.AUDIT_LOG=INFO, AUDIT_LOGFILE log4j.logger.KAFKA=INFO log4j.logger.org.apache.KAFKA=INFO log4j.logger.KAFKA=DEBUG log4j.logger.org.apache.KAFKA=DEBUG log4j.logger.KAFKA=FATAL log4j.logger.org.apache.KAFKA=FATAL log4j.logger.KAFKA=WARN log4j.logger.org.apache.KAFKA=WARN log4j.logger.KAFKA=ERROR log4j.logger.org.apache.KAFKA=ERROR

instead of log4j.rootLogger=ERROR, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG, ERROR_LOGFILE log4j.logger.AUDIT_LOG=INFO, AUDIT_LOGFILE

At the end of the log4j.conf file add the appender

log4j.appender.KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender log4j.appender.KAFKA.brokerList=yourkafkaqueue:9092 log4j.appender.KAFKA.topic=log_raw log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout log4j.appender.KAFKA.layout.ConversionPattern=DATE:%d{yyyy-MM-dd HH:mm:ss.SSS};; HOST:test.com;; SEVERITY:%p;; MSG:%c{1}:%L - %m;;

in /EI/lib put the following jar -kafka-clients-1.1.1.jar -kafka-log4j-appender-1.1.1.jar

The error at the beginning of EI/bin/integrator.sh Using Java memory options: -Xms256m -Xmx1024m Unexpected problem occured during version sanity check Reported exception: java.lang.NullPointerException at org.slf4j.LoggerFactory.versionSanityCheck(LoggerFactory.java:267) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:126) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at org.apache.kafka.clients.CommonClientConfigs.(CommonClientConfigs.java:32) at org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:305) at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:303) at org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:243) at org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:237) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172) at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66) at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72) at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.atomikos.logging.LoggerFactory.(LoggerFactory.java:23) at com.atomikos.transactions.internal.AtomikosActivator.(AtomikosActivator.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at org.eclipse.osgi.framework.internal.core.AbstractBundle.loadBundleActivator(AbstractBundle.java:167) at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:679) at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381) at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:390) at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1176) at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:559) at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:544) at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:457) at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:243) at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:438) at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:1) at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:340) No org.slf4j.impl.StaticLoggerBinder found in ClassPath, trying with log4j...

Jean-Marc-Henry commented 6 years ago

Is there any interaction with this ticket : Remove unnecessary versions of jars from EI pack #2061

prabushi commented 5 years ago

@Jean-Marc-Henry We tried with WSO2 EI 6.4.0 in Ubuntu environment and could reproduce this issue. We will look into this issue for our future release.

Jean-Marc-Henry commented 5 years ago

Hello All, Could you confirm it will be corrected in 6.5 ? Many thanks for all.

Jean-Marc-Henry commented 5 years ago

Hello All, The bug is always here in 6.5.0 m6. Please could you do something for us ?

Jean-Marc-Henry commented 5 years ago

Hello All, The bug is always here in 6.5 You told me that it would be corrected with this release. Please help i can't change version.

Jean-Marc-Henry commented 5 years ago

Hello all, We find that in the pom.xml in the sl4j-api-1.7.21 and in sl4j-api-1.7.25 there are :

....
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>animal-sniffer-maven-plugin</artifactId>
        <configuration>
          <!-- Signatures cannot be determined and will error unless excluded.  This is isolated to
               code otherwise already marked for removal in the module artifact. -->
          <ignores>
            <ignore>org.slf4j.impl.StaticMDCBinder</ignore>
            <ignore>org.slf4j.impl.StaticLoggerBinder</ignore>
            <ignore>org.slf4j.impl.StaticMarkerBinder</ignore>
          </ignores>
        </configuration>
....

Please could you give us more information, or what to do ?

image

@Nirothipan Any information ? Many thanks in advance for your help.

Jean-Marc-Henry commented 5 years ago

Hello All, It works with the 6.6 Snapshot we took on friday!

Many thanks, we will be able to start our migration!

souradeep07 commented 4 years ago

We are facing similar issue on cdh 6.3.3. Can anybody help with some solution?