search

chiwanpark / flume-ng-redis

Redis extension for Flume NG
26 stars 20 forks source link

Sink/Source Usage #11

Open ghost opened 9 years ago

ghost commented 9 years ago

Can elaborate on how to use the the Redis Sink and the Redis source. Am I suppose to modify the config properties file, or the java file and then package with mvn ? Thanks!

chiwanpark commented 9 years ago

Hi. This plugin helps you transfer data from a flume agent to the other flume agent using Redis. Here is a sample structure with this plugin.

               Sink    Source
A Flume agent --- Redis --- A Flume agent

As I wrote in documentation, You can build the plugin package using mvn package -DskipTests. The properties files are for configuration of each flume agent (sink/source).

In example configuration, a flume agent receives data from socket (netcat source), and send to redis using this plugin. Another flume agent receives data from redis using redis source and print data to console.

I hope to help you understand about this plugin. If you have another question, comment to this thread anytime.

ghost commented 9 years ago

Chwin, Thanks for your reply. Sorry I think I wasn’t clear in my question.

I am interested in how the data feed looks like when the flume source ingests that data as in the NetCat Source.

does the data have to be in CSV format, if not, then what format. Is there a serializer to split the event body into values with assigned keys ?

If this is still no clear, please notify me first, and I will put an example in.

Thanks agian!

chiwanpark commented 9 years ago

Hi. Currently, this plugin does not support custom format of data. The plugin supports only raw string data now. Flume has a interface called EventSerializer for dealing with this issue. I will add support for this interface. (or Could you add this feature and send a pull request?)

ghost commented 9 years ago

I am willing to work on the EventSerializer. It might take me a while though to implement a functional and configurable serializer as I am new to programming and development. Thanks again.

fenggolang commented 8 years ago

When I have modified the /flume-ng-redis/conf/example-SubscribeDrivenSource.properties of agent.sources.redis.redisHost=192.168.75.128,next,I am running testSubscribe of RedisSubscribeDrivenSourceTest, The console happen redis

chiwanpark commented 8 years ago

Hi @wpengcheng, example configuration files don't affect unit tests. The unit tests uses Redis on localhost (see this).