Implement Kafka plugins as separate npm package

[megastef]

We got reports that the integrated Kafka plugins try to install native snappy compression library. A check shows that native packages are declared as optional packages nevertheless, the npm installation in a windows container did fail with node-gyp errors.

To avoid setup issues with native packages, I would like to suggest moving Kafka plugins to separate npm packages.

@fbalicchia would you allow that we use your code to create the new npm packages? Or could you provide kafka input and output plugins as separate npm packages e.g. "logagent-input-kafka" and "logagent-output-kafka" ?

[fbalicchia]

Hi @megastef , honestly I never try it on Windows console. Externalize module seems a good solution, that say you are free to use my code as you want but provide kafka input and output plugins as separate npm packages I would not mind

[megastef]

Kafka input plugin on npm: https://www.npmjs.com/package/logagent-input-kafka CC: @fbalicchia

[fbalicchia]

Thanks @megastef ! I'll prepare logagent-out-kafka ASAP if is ok for you. From logagent point of view is necessary a cleanup for remove old plugin. Can I do it ?, or you prefer to do it ?

[megastef]

@fbalicchia help for logagent-output-kafka is very much appreciated. I could clean up the Logagent repository and documentation once we are done with the new npm packages.

Please note changes in https://github.com/megastef/logagent-input-kafka/commits/master

• added Travis test
• added connection retry "forever" in connection settings for kafka client
• check mandatory config values to avoid LA crash by kafka-node errors

I'll add you to logagent-input-kafka repository...

[fbalicchia]

Hi, have start logagent-output-kafka but I don't understard how to test it locally without publish npm and using logagent master. Could you please address me ?

I've used logagent-input-kafka following readMe and It's works for me. later I'll try to understand why in travis env fails... At the moment following log I can't suggest something useful

[megastef]

1) You could simulate logagent plugin env by passing config and event emitter and then call plugin.start. Then generate data, by emitting a 'data.raw' event. See e.g. https://github.com/megastef/logagent-tcp-input/blob/master/test.js

If you adjust above example, your config object should have following structure (equals to the data structure your plugin expects):

var config = {
      output: {
         kafka: {
           module: logagent-output-kafka,
       kafkaHost: localhost,
           topic: test
         }
      }
    }
 }

2) Alternative: Install Logagent and then install the plugin from the local development folder:

npm i -g @sematext/logagent
# install your plugin from local dev folder
npm i -g .
# run Logagent with your plugin config
logagent --config kafka-output.yml

The first approach is better for local debugging or could be extended for automatic testing with mocha. The second approach is good for a final integration test with Logagent.

[megastef]

Git repository for Kafka output plugin: https://github.com/fbalicchia/logagent-output-kafka

[fbalicchia]

Hi Kafka output plugin is ready, is not released yet, my idea is to to transfer ownership to @megastef cause in the near future I can't guarantee my contribute, that not mean that will be abandoned but I'll not be much responsive.

[megastef]

@fbalicchia Thank you very much! - I will have a look this evening. I could publish the package on npm and watch issues. If you don't mind transfer ownership to @sematext - in that way we would see issues in our GitHub slack channel.

@fbalicchia are you interested in writing a blog post about Apache Kafka & Logagent? I've never seen such an easy way to create producers and consumers and I think it could be of interest for other DevOps people.

[fbalicchia]

I try to transfert logagent-output-kafka ownership to Sematext report but I haven't write access grant to repo. Other approach can be uploaded it in Sematext repo. tell me how you want continue np.

Yes, I'm pleasure in write a blog post, about Kafka and logAgent. I'm agree that it is very simple write producer and consumer, that was the Main goal of logagent I think. With introduction of these plugins IMHO is possible to add log buffer between different agent to improve flexibility.

I start to write it soon, when I have something I'll send you a preview for a review and any suggestion are welcome.

Is this a correct channel for these kind of things ?

[megastef]

1) +1 for a blog post. We share drafts via google docs. Contact for e-mail: stefan.thies@sematext.com my skype ID is seti123 - if you like to chat...

2) Repository ownership: If you transfer ownership to 'megastef', i could try to transfer the repo ownership to 'sematext'.

[megastef]

Moved overnership of megastef/logagent-input-kafka to sematext: https://github.com/sematext/logagent-input-kafka

[megastef]

We published fbalicchia/logagent-output-kafka on npm: https://www.npmjs.com/package/logagent-output-kafka. Permissions for publishing are granted to @megastef, @sematext and @fbalicchia

[fbalicchia]

Hi @megastef I've Transfer ownership of logagent-output. On npm repo there is a wrong github reference ;-) it points to input not to output ;-) I try to fix it but i think that I haven't permission

[megastef]

@fbalicchia I fixed URL in fbalicchia/logagent-output-kafka and published new npm package: https://www.npmjs.com/package/logagent-output-kafka. Why don't you have access to your own repository? I can't find megastef/logagent-output-kafka or sematext//logagent-output-kafka - it seems you did not transfer ownership (but I have permissions to change fbalicchia/logagent-output-kafka).

[fbalicchia]

@megastef I've finish post. here a preview: link Any suggestion are welcome . thanks again for opportunity

[megastef]

Thank you @fbalicchia - we close this issue.

NPM Packages: https://www.npmjs.com/package/logagent-output-kafka https://www.npmjs.com/package/logagent-input-kafka

Documentation: http://sematext.com/docs/logagent/input-plugin-kafka/ http://sematext.com/docs/logagent/output-plugin-kafka/