search

heyrogs / BigData

bigdata learn source by self
0 stars 0 forks source link

日志收集flume #4

Open heyrogs opened 6 years ago

heyrogs commented 6 years ago

核心:

三大组件: Source：数据源收集 channel: 管道(缓存) Sink: 输出

功能: 收集、聚合、移动

使用flume的关键是写配置文件 A)配置source B)配置Channel C)配置Sink D)把以上三个组件串起来

以下内容存放在$FLUME_HOME/conf/example.conf文件里

a1: agent 名称

agent 配置定义

agent 配置定义

a1.sources=r1 a1.sinkes=k1 a1.channels=c1

source定义 netcat监听端口

a1.sources.r1.type=netcat a1.sources.r1.bind=jiang a1.sources.r1.port=55555

sink

a1.sinks.k1.type=logger

channel

a1.channels.c1.type=memory

将三个组件链接起来

a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

启动agent： flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/example.conf -Dflume.root.logger=INFO,console

heyrogs commented 6 years ago

kafka + flume:

->1 . avro-memory-kafka.conf

avro-memory-kafka.sources = avro-source avro-memory-kafka.sinks = kafka-sink avro-memory-kafka.channels = memory-channel

sources configuration

avro-memory-kafka.sources.avro-source.type = avro avro-memory-kafka.sources.avro-source.bind = ajiang avro-memory-kafka.sources.avro-source.port = 44444

sink

avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink avro-memory-kafka.sinks.kafka-sink.brokerList = ajiang:9092 avro-memory-kafka.sinks.kafka-sink.topic = hello_topic avro-memory-kafka.sinks.kafka-sink.batchSize = 5 avro-memory-kafka.sinks.kafka-sink.requiredAcks = 1

channel

avro-memory-kafka.channels.memory-channel.type = memory

bind

avro-memory-kafka.sources.avro-source.channels = memory-channel avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

-> 2 . exec-memory-avro.conf

exec-memory-avro.sources = exec-source exec-memory-avro.sinks = avro-sink exec-memory-avro.channels = memory-channel

sources

exec-memory-avro.sources.exec-source.type = exec exec-memory-avro.sources.exec-source.command = tail -F /home/logs/1.txt exec-memory-avro.sources.exec-source.shell = /bin/sh -c

sink

exec-memory-avro.sinks.avro-sink.type = avro exec-memory-avro.sinks.avro-sink.hostname = ajiang exec-memory-avro.sinks.avro-sink.port = 44444

channel

exec-memory-avro.channels.memory-channel.type = memory

exec-memory-avro.sources.exec-source.channels = memory-channel exec-memory-avro.sinks.avro-sink.channel = memory-channel

启动flume: flume-ng agent --name avro-memory-kafka --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf -Dflume.root.logger=INFO,console

flume-ng agent --name exec-memory-avro --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-avro.conf -Dflume.root.logger=INFO,console