What is TD-Agent

[Hearen]

Fluentd is a cross platform open-source data collection software project originally developed at Treasure Data. It is written primarily in the Ruby programming language.

Fluentd v1.0 is available on Linux, Mac OSX and Windows. Also, Treasure Data packages it as Treasure Agent (td-agent) for RedHat/CentOS and Ubuntu/Debian and Windows

Fluentd is a Big Data tool for semi- or un-structured data sets. Like Apache Kafka, it analyzes event logs, application logs, and clickstreams. According to Suonsyrjä and Mikkonen, the "core idea of Fluentd is to be the unifying layer between different types of log inputs and outputs.", Fluentd is available on Linux, Mac OSX, and Windows.

Fluentd was one of the data collection tools recommended by Amazon Web Services in 2013, when it was said to be similar to Apache Flume or Scribe. Google Cloud Platform's BigQuery recommends Fluentd as default real-time data-ingestion tool, and uses Google's customized version of Fluentd, called google-fluentd, as a default logging agent.

References

• https://en.wikipedia.org/wiki/Fluentd
• https://www.fluentd.org/download

[Hearen]

What

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

Unified Logging Layer

Fluentd decouples data sources from backend systems by providing a unified logging layer in between.

This layer allows developers and data analysts to utilize many types of logs as they are generated. Just as importantly, it mitigates the risk of "bad data" slowing down and misinforming your organization.

A unified logging layer lets you and your organization make better use of data and iterate more quickly on your software.

Features

• Unified Logging with JSON
• Pluggable Architecture (Our 500+ community-contributed plugins connect dozens of data sources and data outputs.)
• Minimum Resources Required (Fluentd is written in a combination of C language and Ruby, and requires very little system resource. The vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core. If you have tighter memory requirements (-450kb), check out Fluent Bit, the lightweight forwarder for Fluentd.)
• Built-in Reliability (Fluentd supports memory- and file-based buffering to prevent inter-node data loss. Fluentd also supports robust failover and can be set up for high availability.)

In one word, td-agent is a stable distribution package of Fluentd.

References

• https://www.fluentd.org/architecture
• https://www.fluentd.org/why
• https://www.fluentd.org/blog/unified-logging-layer
• https://www.fluentd.org/faqs

[Hearen]

Catch Up

monitor each stage first and then improve its reliability as the ideal goal

Which part for now shall we start our work? (I know there are different types of configurations: td-agent, prometheus and environments without them);

for now we start from fluentd only (the log of td-agent itself to analyze the status of fluentd);

• for prometheus, it will be configured only in environment scope;
• td-agent as forwarder has to be configured for each instance (basically one to one) which collect logs (also generates logs itself) and requires the log to be uploaded to S3;

• td-agent for aggregator (RC -> ec2; grafana has some statistical graphs) will be configured for each environment (many to one);

  td-agent working details in our case following: forwarder collect in -> forwarder processing -> forwarder pushing out -> aggregator receiving -> aggregator pushing up -> S3 aggregator;

Options

• When Prometheus configured, we are planning to based on the following prometheus statistics to trace down the issues: metrics data, queue length, queue buffer size, retry count, buffer length, wait time, catalina log last modified time, forwarder/aggregator buffer size;

• When Prometheus not configured but td-agent configured, we can monitor fluentd by the log of its own in LV directly; in this way, we have to first manually analyze the logs ourselves and then introduce auto-analyzer to identify the issues;

Questions

• What are the target environment? all of them included? (single tenant, multi-tenant; native, conversion; develop, staging, production and others)?
• Are we to use only the td-agent logs to trace back the status and achieve the monitoring? no other options will be considered? built-in plugins?
• Just eye-checking the logs collected in LV seems not quite effective and reliable, do have any other idea to carry out the monitor?

Monitoring

• monitor td-agent metrics including built-in plugin, prometheus, and datadog
• monitor td-agent including process, port and debug port

Usages

vim /etc/td-agent/config/forwarder/td-agent-forwarded.conf 
sudo less /var/log/td-agent/td-agent.log.1

# adding DAEMON_ARGS=-vv to /etc/default/td-agent to enable verbose logging

References

• https://support.treasuredata.com/hc/en-us/articles/360000687108-Overview-of-Server-Side-Agent-td-agent-

[Hearen]

Integrated

Java

http://central.maven.org/maven2/com/treasuredata/

<dependencies>
  ...
  <dependency>
    <groupId>com.treasuredata</groupId>
    <artifactId>td-logger</artifactId>
    <version>${logger.version}</version>
  </dependency>
  ...
</dependencies>

References

• https://support.treasuredata.com/hc/en-us/articles/360000677507-Server-Side-Agent-with-Java-Apps
• https://support.treasuredata.com/hc/en-us/articles/360000676827
• https://support.treasuredata.com/hc/en-us/articles/360000676967
• https://docs.fluentd.org/v1.0/articles/quickstart