moooofly
commented
5 years ago Overview of Server-Side Agent (td-agent)
Arm Treasure Data provides Server-Side Agent called Treasure Agent (
td-agent), to collect server-side logs and events. You can continuously import data usingtd-agent.
td-agent 是 Arm Treasure Data 公司提供的、用于收集服务器侧日志和事件的 agent ；
Logs Are Streams, Not Files
Logs are usually rotated on an hourly or daily basis based on time or size. This system quickly produces many large log files that need to be batch imported for further analysis. This is an outdated approach. Logs are better treated as continuously generated STREAMS as opposed to files.
旧式日志系统会将 logs 按照 小时 或 每天 的维度、基于 time 或 size 进行 rotate ；而更现代的观念是：应该将 logs 看做持续生成的 流 而不是 文件；
"Server daemons (such as PostgreSQL or Nginx) and applications (such as a Rails or Django app) sometimes offer a configuration parameter for a path to the program’s logfile. This can lead us to think of logs as files. But a better conceptual model is to treat logs as time-ordered streams..." - Logs Are Streams, Not Files Adam Wiggins, Heroku co-founder.
td-agent, a data collection daemon, is used to import data continuously to Treasure Data. Although bulk-import is supported, we recommend importing your data continuously viatd-agent.
td-agent 既支持 bulk-import 也支持流式导入；
What is Treasure Agent?
td-agent is a data collection daemon. It collects logs from various data sources and uploads them to Treasure Data.
Treasure Agent and Fluentd
Use Treasure Agent 3, fluentd v0.14 (v1.0) series. We are deprecating Treasure Agent 2, fluentd v0.12 series.
推荐 Treasure Agent 3 + fluentd v0.14 (v1.0) 组合
How to install Treasure Agent?
Ubuntu & Debian
# 16.04 Xenial (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent3.sh | sh
# 14.04 Trusty
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
# 12.04 Precise
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent3.sh | sh
# Debian Stretch (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-stretch-td-agent3.sh | sh
# Debian Jessie (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-jessie-td-agent3.sh | sh
# Debian Squeeze (64-bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | shSet up td-agent
After installing
td-agent, you can modify your config file. The file can be found in/etc/td-agent/td-agent.conf.
（配置文件略）
Restart the td-agent service.
# Linux
$ sudo /etc/init.d/td-agent restart
# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plistAgent Overhead
The td-agent resource consumption is roughly:
- Resident memory (actual RAM used): 50MB
- CPU Runtime: less than 2% of averaged runtime + your workload
- Disk:
- Linux 120MB + your file buffer (configurable)
If you think td-agent is slow, see “5 Tips to Optimize Fluentd Performance”.
Files Installed by the Packages
| Resource | Location | Notes |
|---|---|---|
| Config Directory | /etc/td-agent/ | |
| Config File | /etc/td-agent/td-agent.conf | This config will be picked-up by the startup script |
| Startup Script | /etc/init.d/td-agent | |
| Log Directory | /var/log/td-agent/ | |
| Plugin Directory | /etc/td-agent/plugin/ | Your custom plugins go here. |
| Ruby Interpreter | /opt/td-agent/embedded/bin/ruby | Ruby v2.1 is bundled with the package. |
| Rubygems | /usr/sbin/td-agent-gem | Bundled rubygems to install fluentd plugins. For example: /usr/sbin/td-agent-gem install fluent-plugin-mongo |
| jemalloc | /opt/td-agent/embedded/lib/libjemalloc.so | jemalloc is bundled together to avoid memory fragmentation. It is loaded by default in the startup script. |
Supervision, Privileges and Network Ports
When td-agent starts, it launches 2 processes: master and slave. The master process is managing the life cycle of slave process, and slave process handles actual log collection.
Both processes run as the td-agent user under td-agent group, and all forked subprocesses run as the same. This applies to any system call initiated by td-agent as well. The agent configuration resides at /etc/td-agent/td-agent.conf. All configurations must be readable by td-agent.
The following ports are open depending on your input.
- in_tail: nothing
- in_forward: tcp/24224, udp/24224
- in_unix: /var/run/td-agent/td-agent.sock
Debugging
If you are having issues, add the following line to /etc/default/td-agent to enable verbose logging:
DAEMON_ARGS=-vvAfter that, restart the daemon. You can now find more verbose logs in /var/log/td-agent.log
High-Availability Configurations and Monitoring
For high-traffic websites, we recommend using a high availability configuration for td-agent. Monitoring the daemon is also important.
td-agent is fully open-sourced under the fluentd project.
公司文档和项目：
文章：
插件（下面的内容主要是 http output 相关的）：