search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

A few questions re log-courier and 2.x #257

Closed sysmonk closed 8 years ago

sysmonk commented 9 years ago

Hi @driskell,

Wanted to ask a few questions that i'm interested in:

  1. How's the work on log-courier 2 going? Any estimates? I've noticed that the 2.0 branch hadn't had a commit in 3 months now :( Maybe i can help somehow? I'm not a go-er, but would gladly test stuff out!
  2. Are there any plans to support chaining the codecs? My servers generated more than a billion log entries in the last 24h that i drop on the LS side. I'd just use the filter codec, but i'm already using the multiline codec :(
  3. I really love the project, so if there's anything else i could help with - let me know!
driskell commented 9 years ago

Hello!

  1. As mentioned in #250, I've been separating the Ruby library and Logstash plugins into their own repositories, and mostly working on new plugin architecture for better Logstash stability memory wise. I'm now picking back on Log Courier 2.0 though as the load balancing and multiple codecs are a lot more useful. I'll push up something soon with the repository cleaned up and all the 2.0 bits so it can be tested and tried out. The load balancing is working but just needs some maturing to cope with failures better.
  2. Yep that's in the 2.0 branch
  3. Testing is probably one thing lots of people can help with. So I'll do my best to get all the 2.0 work into master as soon as I can so we can see where it needs some work and get it out the door!
driskell commented 8 years ago

I finally found some time and I'm starting to work through on 2.0, everything is in the master branch. There's bits I still need to push on the load balancing strategies and get backwards compatibility mode (method of random and single connection instead of multiple.) Also lots to do on tests :) Planning to deploy into a live environment soon.

driskell commented 8 years ago

2.0 is generally stable now. I'm now getting the new API embedded to improve lc-admin. Then some back offs on network errors etc. If you're able to test it that'll be great.

sysmonk commented 8 years ago

Nice! Thanks for the work!

A few questions

  1. where do i get log-courier 2.0? The git 'master' branch has last commit 18 days ago, and there is no '2.x' tag
  2. do i need to update log-courier logstash plugin? Can it work with the old plugin and both log-courier versions (1.8 and 2.x) ?
  3. what are the incompatible changes between 1.8 and 2.x? I remember seeing something about multiline, but maybe my memory is bad :)

I can try testing next week with a few moderately loaded clients ( 400-1000 logs per second per box ).

driskell commented 8 years ago

Master branch is 2.0. I've been working on the new API locally until I get it tied in everywhere this the 18 days. There's no tag just yet - I plan to tag it as a beta once I finish all the features including API.

sysmonk commented 8 years ago

Thanks for answer @driskell. But you only answered question 1), question 2 and 3 are unanswered :)

driskell commented 8 years ago

Whoops!

  1. Yes current plugin is forwards compatible! No need for any change on Logstash side. 2.0 plugin isn't ready yet either
  2. Change log is at https://github.com/driskell/log-courier/blob/master/CHANGELOG.md but I did miss a few things I bet. I can see already I missed that "codecs" configuration is now an array in the config so you can do both filtering AND Multiline
driskell commented 8 years ago

I'm in the process of releasing 2.0 to RPM and DEB and will tag on GitHub and switch branches in the next few days. So far it is running great for me, with a few minor glitches which I've opened issues for, and plan to fix imminently with a 2.1 release.