Open Hearen opened 5 years ago
Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
Fluentd decouples data sources from backend systems by providing a unified logging layer in between.
This layer allows developers and data analysts to utilize many types of logs as they are generated. Just as importantly, it mitigates the risk of "bad data" slowing down and misinforming your organization.
A unified logging layer lets you and your organization make better use of data and iterate more quickly on your software.
In one word, td-agent is a stable distribution package of Fluentd.
monitor each stage first and then improve its reliability as the ideal goal
Which part for now shall we start our work? (I know there are different types of configurations: td-agent, prometheus and environments without them);
for now we start from fluentd only (the log of td-agent itself to analyze the status of fluentd);
td-agent for aggregator (RC -> ec2; grafana has some statistical graphs) will be configured for each environment (many to one);
td-agent working details in our case following: forwarder collect in -> forwarder processing -> forwarder pushing out -> aggregator receiving -> aggregator pushing up -> S3 aggregator;
When Prometheus configured, we are planning to based on the following prometheus statistics to trace down the issues: metrics data, queue length, queue buffer size, retry count, buffer length, wait time, catalina log last modified time, forwarder/aggregator buffer size;
When Prometheus not configured but td-agent configured, we can monitor fluentd by the log of its own in LV directly; in this way, we have to first manually analyze the logs ourselves and then introduce auto-analyzer to identify the issues;
vim /etc/td-agent/config/forwarder/td-agent-forwarded.conf
sudo less /var/log/td-agent/td-agent.log.1
# adding DAEMON_ARGS=-vv to /etc/default/td-agent to enable verbose logging
http://central.maven.org/maven2/com/treasuredata/
<dependencies>
...
<dependency>
<groupId>com.treasuredata</groupId>
<artifactId>td-logger</artifactId>
<version>${logger.version}</version>
</dependency>
...
</dependencies>
Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.
Fluentd v1.0 is available on Linux, Mac OSX and Windows. Also, Treasure Data packages it as Treasure Agent (td-agent) for RedHat/CentOS and Ubuntu/Debian and Windows
References