search

awslabs / amazon-kinesis-agent

Continuously monitors a set of log files and sends new data to the Amazon Kinesis Stream and Amazon Kinesis Firehose in near-real-time.
Other
352 stars 218 forks source link

No Log output when running with start-aws-kinesis-agent #243

Open wxc267 opened 2 years ago

wxc267 commented 2 years ago

Currently, I am planning running two aws-kinesis-agent in the docker for sending data into two endpoints in different region. Thus, I am starting aws-kinesis-agent by using the executable file start-aws-kinesis-agent.

However, different from running aws-kinesis-agent by systemctl, while running start-aws-kinesis-agent, I found that /var/log/aws-kinesis-agent is empty (no runnning logs).

Could you tell me where the log is? I am running the agent by command: sudo /usr/bin/start-aws-kinesis-agent -c /etc/aws-kinesis/agent1.json & sudo /usr/bin/start-aws-kinesis-agent -c /etc/aws-kinesis/agent2.json &

desp0916 commented 2 years ago

I have the same problem. After installing aws-kinesis-agent-2.0.6-1.amzn2.noarch.rpm on EC2, to debug the problem, I started the agent manually by using the following command:

/usr/lib/jvm/java/bin/java -server -Xms32m -Xmx512m -Dlog4j.configurationFile=file:///etc/aws-kinesis/log4j.xml -XX:OnOutOfMemoryError="/bin/kill -9 %p" -cp /usr/share/aws-kinesis-agent/lib:/usr/share/aws-kinesis-agent/lib/AWSKinesisStreamingDataAgent-1.1.jar:/usr/share/aws-kinesis-agent/lib/aws-java-sdk-cloudwatch-1.11.700.jar:/usr/share/aws-kinesis-agent/lib/aws-java-sdk-core-1.11.700.jar:/usr/share/aws-kinesis-agent/lib/aws-java-sdk-ec2-1.11.700.jar:/usr/share/aws-kinesis-agent/lib/aws-java-sdk-kinesis-1.11.700.jar:/usr/share/aws-kinesis-agent/lib/aws-java-sdk-sts-1.11.700.jar:/usr/share/aws-kinesis-agent/lib/commons-cli-1.2.jar:/usr/share/aws-kinesis-agent/lib/commons-codec-1.6.jar:/usr/share/aws-kinesis-agent/lib/commons-lang3-3.4.jar:/usr/share/aws-kinesis-agent/lib/commons-logging-adapters-1.1.jar:/usr/share/aws-kinesis-agent/lib/commons-logging-api-1.1.jar:/usr/share/aws-kinesis-agent/lib/guava-28.2-jre.jar:/usr/share/aws-kinesis-agent/lib/httpclient-4.5.1.jar:/usr/share/aws-kinesis-agent/lib/httpclient-cache-4.5.1.jar:/usr/share/aws-kinesis-agent/lib/httpcore-4.4.3.jar:/usr/share/aws-kinesis-agent/lib/httpcore-ab-4.4.3.jar:/usr/share/aws-kinesis-agent/lib/httpcore-nio-4.4.3.jar:/usr/share/aws-kinesis-agent/lib/httpmime-4.5.1.jar:/usr/share/aws-kinesis-agent/lib/jackson-annotations-2.10.3.jar:/usr/share/aws-kinesis-agent/lib/jackson-core-2.10.3.jar:/usr/share/aws-kinesis-agent/lib/jackson-databind-2.10.3.jar:/usr/share/aws-kinesis-agent/lib/jackson-dataformat-cbor-2.10.3.jar:/usr/share/aws-kinesis-agent/lib/jackson-dataformat-xml-2.10.3.jar:/usr/share/aws-kinesis-agent/lib/jcommander-1.48.jar:/usr/share/aws-kinesis-agent/lib/joda-time-2.8.2.jar:/usr/share/aws-kinesis-agent/lib/jsr305-3.0.1.jar:/usr/share/aws-kinesis-agent/lib/log4j-1.2-api-2.13.2.jar:/usr/share/aws-kinesis-agent/lib/log4j-api-2.13.2.jar:/usr/share/aws-kinesis-agent/lib/log4j-core-2.13.2.jar:/usr/share/aws-kinesis-agent/lib/log4j-slf4j-impl-2.13.2.jar:/usr/share/aws-kinesis-agent/lib/slf4j-api-1.7.12.jar:/usr/share/aws-kinesis-agent/lib/sqlite-jdbc-3.20.1.jar:/usr/share/java/*: com.amazon.kinesis.streaming.agent.Agent

then I got the following exceptions:

ERROR StatusLogger Caught javax.xml.parsers.ParserConfigurationException setting feature http://xml.org/sax/features/external-general-entities to false on DocumentBuilderFactory org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@7770f470: javax.xml.parsers.ParserConfigurationException: Feature 'http://xml.org/sax/features/external-general-entities' is not recognized. javax.xml.parsers.ParserConfigurationException: Feature 'http://xml.org/sax/features/external-general-entities' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.setFeature(Unknown Source) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:204) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:197) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:186) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.(XmlConfiguration.java:89) at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:517) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:492) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:415) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:687) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304) at com.amazon.kinesis.streaming.agent.Agent.main(Agent.java:59) ERROR StatusLogger Caught javax.xml.parsers.ParserConfigurationException setting feature http://xml.org/sax/features/external-parameter-entities to false on DocumentBuilderFactory org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@7770f470: javax.xml.parsers.ParserConfigurationException: Feature 'http://xml.org/sax/features/external-parameter-entities' is not recognized. javax.xml.parsers.ParserConfigurationException: Feature 'http://xml.org/sax/features/external-parameter-entities' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.setFeature(Unknown Source) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:204) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:198) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:186) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.(XmlConfiguration.java:89) at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:517) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:492) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:415) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:687) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304) at com.amazon.kinesis.streaming.agent.Agent.main(Agent.java:59) ERROR StatusLogger Caught javax.xml.parsers.ParserConfigurationException setting feature http://apache.org/xml/features/nonvalidating/load-external-dtd to false on DocumentBuilderFactory org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@7770f470: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/nonvalidating/load-external-dtd' is not recognized. javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/nonvalidating/load-external-dtd' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.setFeature(Unknown Source) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.setFeature(XmlConfiguration.java:204) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.disableDtdProcessing(XmlConfiguration.java:199) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:186) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.(XmlConfiguration.java:89) at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:517) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:492) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:415) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:687) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304) at com.amazon.kinesis.streaming.agent.Agent.main(Agent.java:59) ERROR StatusLogger Error parsing /etc/aws-kinesis/log4j.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.newDocumentBuilder(XmlConfiguration.java:191) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.(XmlConfiguration.java:89) at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:46) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:517) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:492) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:415) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:687) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304) at com.amazon.kinesis.streaming.agent.Agent.main(Agent.java:59) ERROR StatusLogger No logging configuration

desp0916 commented 2 years ago

I have found one solution, change line 129 in /usr/bin/start-aws-kinesis-agent from:

#CLASSPATH="$LIB_DIR":$(find "$LIB_DIR" -type f -name \*.jar | paste -s -d:):"${JAVA_DIR}/*":"${CLASSPATH}"

to

CLASSPATH="$LIB_DIR":$(find "$LIB_DIR" -type f -name \*.jar | paste -s -d:)

Then remember to sudo service aws-kinesis-agent restart, the log will appear.

There is /usr/share/java/log4j.jar on my EC2 instance, it cause the problem.

rpm -qfi /usr/share/java/log4j.jar

Name : log4j Epoch : 0 Version : 1.2.17 Release : 18.amzn2 Architecture: noarch Install Date: Thu 21 Jul 2022 06:07:07 AM UTC Group : Unspecified Size : 519214 License : ASL 2.0 Signature : RSA/SHA256, Mon 14 Feb 2022 09:16:24 PM UTC, Key ID 11cf1f95c87f5b1a Source RPM : log4j-1.2.17-18.amzn2.src.rpm Build Date : Thu 10 Feb 2022 11:01:39 PM UTC Build Host : build.amazon.com Relocations : (not relocatable) Packager : Amazon Linux Vendor : Amazon Linux URL : http://logging.apache.org/log4j Summary : Java logging package Description : Log4j is a tool to help the programmer output log statements to a variety of output targets.