search

jrh3k5 / flume-plugin-maven-plugin

A Maven plugin to generate archives that can be used as plugins in a Flume agent.
Apache License 2.0
0 stars 1 forks source link

Support bundling Flume plugins as a Cloudera parcels #5

Open jrh3k5 opened 10 years ago

jrh3k5 commented 10 years ago

Some consumers of Flume use the Cloudera distribution, which uses a ".parcel" format (which is a .tar.gz with some meta descriptor files) to distribute things like plugins.

Some doc can be found here:

http://blog.cloudera.com/blog/2013/07/one-engineers-experience-with-parcel/

jrh3k5 commented 10 years ago

CDH4 bundles, including bundles for Flume, can be found here:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH-Version-and-Packaging-Information/cdhvd_topic_3.html

jrh3k5 commented 10 years ago

More info: found a Google Groups comment that indicates where Cloudera Flume looks for plugins:

There are two options. If you're running CM 4.5.x (must be at least 4.5.1) you can set CLASSPATH variables in the environment safety valve for flume:

Add jars to FLUME_CLASSPATH Add native libs to FLUME_JAVA_LIBRARY_PATH With CM 4.6 and CDH 4.3, you can set up a Flume plugins.d directory (as described in flume docs) structure and then specify that in the flume configuration.

By default, CM 4.6 configures flume to look at:

/usr/lib/flume-ng/plugins.d

and

/var/lib/flume-ng/plugins.d

So you can stick your plugins under the /var location and they'll get picked up automatically and be used, when you move from parcel to parcel.

Or you can specify a custom location. For example:

/opt/flume.plugins.d/ /opt/flume.plugins.d/sink-a/ /opt/flume.plugins.d/sink-a/lib /opt/flume.plugins.d/sink-a/libext /opt/flume.plugins.d/sink-a/native /opt/flume.plugins.d/sink-b/ /opt/flume.plugins.d/sink-b/lib /opt/flume.plugins.d/sink-b/libext /opt/flume.plugins.d/sink-b/native /opt/flume.plugins.d/source-a/ /opt/flume.plugins.d/source-a/lib /opt/flume.plugins.d/source-a/libext /opt/flume.plugins.d/source-a/native

Then you add /opt/flume.plugins.d to the 'Plugin directories' config