moooofly
commented
5 years ago Overview
At the high level, a simple Beat has two main components:
- a component that collects the actual data, and
- a publisher that sends the data to the specified output, such as Elasticsearch or Logstash.
The publisher is already implemented in libbeat, so you typically only have to worry about the logic specific to your Beat (the code that creates the event and sends it to the publisher). Libbeat also offers common services like configuration management, logging, daemonzing, and Windows service handling, and data processing modules.
The event that you create is a JSON-like object (Go type map[string]interface{}) that contains the collected data to send to the publisher. At a minimum, the event object must contain a @timestamp field and a type field. Beyond that, events can contain any additional fields, and they can be created as often as necessary.
Generating Your Beat
To generate your own Beat, you use the Beat generator available in the beats repo on GitHub.
Python 2 is required (Python 3 will not work).
Run python and specify the path to the Beat generator:
[#529#root@ubuntu-1604 /go/src/github.com/moooofly]$python $GOPATH/src/github.com/elastic/beats/script/generate.py
Beat Name [Examplebeat]: Httpbeat
Your Github Name [your-github-name]: moooofly
Beat Path [github.com/moooofly/httpbeat]:
Firstname Lastname: Fei Sun
[#530#root@ubuntu-1604 /go/src/github.com/moooofly]$自动生成如下内容
[#531#root@ubuntu-1604 /go/src/github.com/moooofly]$ll httpbeat/
total 76
drwxr-xr-x 8 root root 4096 Mar 13 15:32 ./
drwxr-xr-x 19 root root 4096 Mar 13 15:32 ../
drwxr-xr-x 2 root root 4096 Mar 13 15:32 beater/
drwxr-xr-x 2 root root 4096 Mar 13 15:32 cmd/
drwxr-xr-x 2 root root 4096 Mar 13 15:32 config/
-rw-r--r-- 1 root root 0 Mar 13 15:32 CONTRIBUTING.md
drwxr-xr-x 2 root root 4096 Mar 13 15:32 docs/
-rw-r--r-- 1 root root 362 Mar 13 15:32 .editorconfig
-rw-r--r-- 1 root root 56 Mar 13 15:32 .gitignore
-rw-r--r-- 1 root root 552 Mar 13 15:32 LICENSE.txt
-rw-r--r-- 1 root root 3228 Mar 13 15:32 magefile.go
-rw-r--r-- 1 root root 191 Mar 13 15:32 main.go
-rw-r--r-- 1 root root 599 Mar 13 15:32 main_test.go
-rw-r--r-- 1 root root 326 Mar 13 15:32 make.bat
-rw-r--r-- 1 root root 1268 Mar 13 15:32 Makefile
drwxr-xr-x 2 root root 4096 Mar 13 15:32 _meta/
-rw-r--r-- 1 root root 134 Mar 13 15:32 NOTICE.txt
-rw-r--r-- 1 root root 2484 Mar 13 15:32 README.md
drwxr-xr-x 3 root root 4096 Mar 13 15:32 tests/
-rw-r--r-- 1 root root 862 Mar 13 15:32 .travis.yml
[#532#root@ubuntu-1604 /go/src/github.com/moooofly]$You now have a raw template of the Beat, but you still need to fetch dependencies and set up the Beat.
Fetching Dependencies and Setting up the Beat
First you need to install the following tools:
- Python
- virtualenv --
apt install virtualenv
To fetch dependencies and set up the Beat, run:
cd ${GOPATH}/src/github.com/moooofly/httpbeat
make setup错误一：Makefile 编写的问题，执行第一次时，环境变量 ES_BEATS 为空值，第二次执行时就有值了
[#536#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$make setup
mkdir -p vendor/github.com/elastic
cp -R /go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git vendor/github.com/elastic/beats/x-pack
mkdir -p vendor/github.com/magefile
cp -R /go/src/github.com/elastic/beats/vendor/github.com/magefile/mage vendor/github.com/magefile
git init
Initialized empty Git repository in /go/src/github.com/moooofly/httpbeat/.git/
make: *** No rule to make target 'update', needed by 'setup'. Stop.
[#537#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$错误二：实现没有安装 virtualenv
[#543#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$make setup
mkdir -p vendor/github.com/elastic
cp -R /go/src/github.com/elastic/beats vendor/github.com/elastic/
rm -rf vendor/github.com/elastic/beats/.git vendor/github.com/elastic/beats/x-pack
mkdir -p vendor/github.com/magefile
cp -R /go/src/github.com/elastic/beats/vendor/github.com/magefile/mage vendor/github.com/magefile
git init
Reinitialized existing Git repository in /go/src/github.com/moooofly/httpbeat/.git/
bash: virtualenv: command not found
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:247: recipe for target 'python-env' failed
make: *** [python-env] Error 127
[#544#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$The Beat now contains the basic config file, httpbeat.yml, and template files. The Beat is "complete" in the sense that you can compile and run it. However, to make it functionally complete, you need to add your custom logic (see The Beater Interface), along with any additional configuration parameters that your Beat requires.
到此，已经得到了一个可以编译运行的完整模板文件，接下来只要编写逻辑代码就可以了；
Building and Running the Beat
To compile the Beat, make sure you are in the Beat directory and run:
makeNow run the Beat:
./httpbeat -e -d "*"The command automatically loads the default config file, httpbeat.yml, and sends debug output to the console.
You can stop the Beat by pressing Ctrl+C.
[#565#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$./httpbeat -e -d "*"
2019-03-13T17:02:47.387+0800 INFO instance/beat.go:616 Home path: [/go/src/github.com/moooofly/httpbeat] Config path: [/go/src/github.com/moooofly/httpbeat] Data path: [/go/src/github.com/moooofly/httpbeat/data] Logs path: [/go/src/github.com/moooofly/httpbeat/logs]
2019-03-13T17:02:47.387+0800 DEBUG [beat] instance/beat.go:653 Beat metadata path: /go/src/github.com/moooofly/httpbeat/data/meta.json
2019-03-13T17:02:47.387+0800 INFO instance/beat.go:623 Beat UUID: ac05795d-8f29-4321-8fd4-3210da68b2db
2019-03-13T17:02:47.387+0800 DEBUG [seccomp] seccomp/seccomp.go:109 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2019-03-13T17:02:47.387+0800 INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2019-03-13T17:02:47.387+0800 INFO [beat] instance/beat.go:936 Beat info {"system_info": {"beat": {"path": {"config": "/go/src/github.com/moooofly/httpbeat", "data": "/go/src/github.com/moooofly/httpbeat/data", "home": "/go/src/github.com/moooofly/httpbeat", "logs": "/go/src/github.com/moooofly/httpbeat/logs"}, "type": "httpbeat", "uuid": "ac05795d-8f29-4321-8fd4-3210da68b2db"}}}
2019-03-13T17:02:47.387+0800 INFO [beat] instance/beat.go:945 Build info {"system_info": {"build": {"commit": "58c49c40f3c770b5e5afe820cb69b041cbb51241", "libbeat": "6.6.3", "time": "2019-03-13T08:53:13.000Z", "version": "6.6.3"}}}
2019-03-13T17:02:47.387+0800 INFO [beat] instance/beat.go:948 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.11.1"}}}
2019-03-13T17:02:47.388+0800 INFO [beat] instance/beat.go:952 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-03-03T07:50:26+08:00","containerized":false,"name":"ubuntu-1604","ip":["127.0.0.1/8","::1/128","10.0.2.15/24","fe80::14:68ff:fe3e:8447/64","11.11.11.12/24","172.17.0.1/16"],"kernel_version":"4.4.0-141-generic","mac":["02:14:68:3e:84:47","08:00:27:bf:25:a3","02:42:8a:36:f2:54"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.3 LTS (Xenial Xerus)","major":16,"minor":4,"patch":3,"codename":"xenial"},"timezone":"CST","timezone_offset_sec":28800,"id":"36048e2ecb0f4be19d57697a529c5d3e"}}}
2019-03-13T17:02:47.389+0800 INFO [beat] instance/beat.go:981 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/go/src/github.com/moooofly/httpbeat", "exe": "/go/src/github.com/moooofly/httpbeat/httpbeat", "name": "httpbeat", "pid": 28551, "ppid": 3472, "seccomp": {"mode":"filter"}, "start_time": "2019-03-13T17:02:46.570+0800"}}}
2019-03-13T17:02:47.389+0800 INFO instance/beat.go:281 Setup Beat: httpbeat; Version: 6.6.3
2019-03-13T17:02:47.389+0800 DEBUG [beat] instance/beat.go:302 Initializing output plugins
2019-03-13T17:02:47.390+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:160 add_cloud_metadata: starting to fetch metadata, timeout=3s
2019-03-13T17:02:47.422+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:192 add_cloud_metadata: received disposition for qcloud after 31.618035ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com on 10.0.2.3:53: no such host, metadata={}]
2019-03-13T17:02:50.391+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:192 add_cloud_metadata: received disposition for ec2 after 3.00082132s. result=[provider:ec2, error=failed requesting ec2 metadata: Get http://169.254.169.254/2014-02-25/dynamic/instance-identity/document: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:192 add_cloud_metadata: received disposition for gce after 3.000963834s. result=[provider:gce, error=failed requesting gce metadata: Get http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:192 add_cloud_metadata: received disposition for az after 3.001104536s. result=[provider:az, error=failed requesting az metadata: Get http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:192 add_cloud_metadata: received disposition for digitalocean after 3.001183266s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get http://169.254.169.254/metadata/v1.json: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.391+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:192 add_cloud_metadata: received disposition for openstack after 3.001291458s. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/instance-id: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2019-03-13T17:02:50.392+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:199 add_cloud_metadata: timed-out waiting for all responses
2019-03-13T17:02:50.392+0800 DEBUG [filters] add_cloud_metadata/add_cloud_metadata.go:163 add_cloud_metadata: fetchMetadata ran for 3.00138628s
2019-03-13T17:02:50.392+0800 INFO add_cloud_metadata/add_cloud_metadata.go:319 add_cloud_metadata: hosting provider type not detected.
2019-03-13T17:02:50.392+0800 DEBUG [processors] processors/processor.go:66 Processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null
2019-03-13T17:02:50.392+0800 INFO elasticsearch/client.go:165 Elasticsearch url: http://localhost:9200
2019-03-13T17:02:50.393+0800 DEBUG [publish] pipeline/consumer.go:137 start pipeline event consumer
2019-03-13T17:02:50.393+0800 INFO [publisher] pipeline/module.go:110 Beat name: ubuntu-1604
2019-03-13T17:02:50.393+0800 INFO instance/beat.go:403 httpbeat start running.
2019-03-13T17:02:50.393+0800 INFO beater/httpbeat.go:37 httpbeat is running! Hit CTRL-C to stop it.
2019-03-13T17:02:50.393+0800 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-03-13T17:02:51.395+0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-13T09:02:51.395Z",
"@metadata": {
"beat": "httpbeat",
"type": "doc",
"version": "6.6.3"
},
"type": "ubuntu-1604",
"beat": {
"name": "ubuntu-1604",
"hostname": "ubuntu-1604",
"version": "6.6.3"
},
"host": {
"containerized": false,
"architecture": "x86_64",
"os": {
"name": "Ubuntu",
"codename": "xenial",
"platform": "ubuntu",
"version": "16.04.3 LTS (Xenial Xerus)",
"family": "debian"
},
"id": "36048e2ecb0f4be19d57697a529c5d3e",
"name": "ubuntu-1604"
},
"counter": 1
}
2019-03-13T17:02:51.395+0800 INFO beater/httpbeat.go:62 Event sent
2019-03-13T17:02:52.395+0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-13T09:02:52.395Z",
"@metadata": {
"beat": "httpbeat",
"type": "doc",
"version": "6.6.3"
},
"type": "ubuntu-1604",
"counter": 2,
"beat": {
"hostname": "ubuntu-1604",
"version": "6.6.3",
"name": "ubuntu-1604"
},
"host": {
"containerized": false,
"architecture": "x86_64",
"name": "ubuntu-1604",
"os": {
"version": "16.04.3 LTS (Xenial Xerus)",
"family": "debian",
"name": "Ubuntu",
"codename": "xenial",
"platform": "ubuntu"
},
"id": "36048e2ecb0f4be19d57697a529c5d3e"
}
}
2019-03-13T17:02:52.395+0800 INFO beater/httpbeat.go:62 Event sent
2019-03-13T17:02:52.396+0800 INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://localhost:9200))
2019-03-13T17:02:52.396+0800 DEBUG [elasticsearch] elasticsearch/client.go:697 ES Ping(url=http://localhost:9200)
2019-03-13T17:02:52.397+0800 DEBUG [elasticsearch] elasticsearch/client.go:701 Ping request failed with: Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-13T17:02:53.395+0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-13T09:02:53.395Z",
"@metadata": {
"beat": "httpbeat",
"type": "doc",
"version": "6.6.3"
},
"type": "ubuntu-1604",
"counter": 3,
"beat": {
"name": "ubuntu-1604",
"hostname": "ubuntu-1604",
"version": "6.6.3"
},
"host": {
"name": "ubuntu-1604",
"id": "36048e2ecb0f4be19d57697a529c5d3e",
"containerized": false,
"architecture": "x86_64",
"os": {
"platform": "ubuntu",
"version": "16.04.3 LTS (Xenial Xerus)",
"family": "debian",
"name": "Ubuntu",
"codename": "xenial"
}
}
}
2019-03-13T17:02:53.396+0800 INFO beater/httpbeat.go:62 Event sent
2019-03-13T17:02:54.235+0800 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-13T17:02:54.235+0800 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://localhost:9200)) with 1 reconnect attempt(s)
2019-03-13T17:02:54.235+0800 DEBUG [elasticsearch] elasticsearch/client.go:697 ES Ping(url=http://localhost:9200)
2019-03-13T17:02:54.235+0800 DEBUG [elasticsearch] elasticsearch/client.go:701 Ping request failed with: Get http://localhost:9200: dial tcp 127.0.0.1:9200: connect: connection refused
2019-03-13T17:02:54.395+0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-13T09:02:54.395Z",
"@metadata": {
"beat": "httpbeat",
"type": "doc",
"version": "6.6.3"
},
"host": {
"containerized": false,
"architecture": "x86_64",
"os": {
"codename": "xenial",
"platform": "ubuntu",
"version": "16.04.3 LTS (Xenial Xerus)",
"family": "debian",
"name": "Ubuntu"
},
"name": "ubuntu-1604",
"id": "36048e2ecb0f4be19d57697a529c5d3e"
},
"type": "ubuntu-1604",
"counter": 4,
"beat": {
"name": "ubuntu-1604",
"hostname": "ubuntu-1604",
"version": "6.6.3"
}
}
2019-03-13T17:02:54.395+0800 INFO beater/httpbeat.go:62 Event sent
2019-03-13T17:02:55.394+0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"@timestamp": "2019-03-13T09:02:55.394Z",
"@metadata": {
"beat": "httpbeat",
"type": "doc",
"version": "6.6.3"
},
"counter": 5,
"beat": {
"hostname": "ubuntu-1604",
"version": "6.6.3",
"name": "ubuntu-1604"
},
"host": {
"name": "ubuntu-1604",
"os": {
"platform": "ubuntu",
"version": "16.04.3 LTS (Xenial Xerus)",
"family": "debian",
"name": "Ubuntu",
"codename": "xenial"
},
"id": "36048e2ecb0f4be19d57697a529c5d3e",
"containerized": false,
"architecture": "x86_64"
},
"type": "ubuntu-1604"
}
2019-03-13T17:02:55.395+0800 INFO beater/httpbeat.go:62 Event sent
^C2019-03-13T17:02:55.849+0800 DEBUG [service] service/service.go:50 Received sigterm/sigint, stopping
2019-03-13T17:02:55.849+0800 DEBUG [publish] pipeline/client.go:148 client: closing acker
2019-03-13T17:02:55.849+0800 DEBUG [publish] pipeline/client.go:150 client: done closing acker
2019-03-13T17:02:55.849+0800 DEBUG [publish] pipeline/client.go:154 client: cancelled 0 events
2019-03-13T17:02:55.851+0800 INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":4}},"total":{"ticks":20,"time":{"ms":32},"value":0},"user":{"ticks":20,"time":{"ms":28}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":5},"info":{"ephemeral_id":"35fbc8ce-486d-4a6b-98c3-14e0161d6d95","uptime":{"ms":8468}},"memstats":{"gc_next":4194304,"memory_alloc":2101912,"memory_total":4043280,"rss":25067520}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":5,"published":5,"retry":2,"total":5}}},"system":{"cpu":{"cores":2},"load":{"1":0.06,"15":0.04,"5":0.06,"norm":{"1":0.03,"15":0.02,"5":0.03}}}}}}
2019-03-13T17:02:55.851+0800 INFO [monitoring] log/log.go:153 Uptime: 8.469085575s
2019-03-13T17:02:55.851+0800 INFO [monitoring] log/log.go:130 Stopping metrics logging.
2019-03-13T17:02:55.851+0800 INFO instance/beat.go:413 httpbeat stopped.
[#566#root@ubuntu-1604 /go/src/github.com/moooofly/httpbeat]$
ref: https://www.elastic.co/guide/en/beats/devguide/6.6/new-beat.html