It's a NodeJS implementation of Logstash.
node-logstash is a tool to collect logs on servers. It allows sending its logs to a central server and to ElasticSearch for indexing.
On top of the elastic search, you can use a specialized interface like kibana to dive into your logs.
When I tried logstash, I had some problems. This project mainly addresses those problems. This version should have:
Moreover, it's written in NodeJS, which is a perfect language for programs with many IO.
node-logstash is compatible with logstash. You can replace a node-logstash node with a logstash one. The data are formatted in the same way to be compatible with logstash UIs.
The architecture is identical to logstash architecture. You have to instantiate plugins with the node-logstash core. There are three type of modules:
A typical node-logstash deployment contains agents to crawl logs and a log server.
On agent, node-logstash is configured with inputs plugins to get logs from your software stack, and one output plugin to send logs to log server (eg. zeromq output plugin).
On log server, logs come through a zeromq input plugin, are processed (fields and timestamps extraction), and send to ElasticSearch.
Please open an issue.
October 25th, 2015.
When I started node-logstash, the ecosystem around logstash and ElasticSearch were almost non-existent. In 2015, the situation is not the same :
So, what is the future of node-logstash?
Node-logstash is production ready and used in production. Installation is a classical node project installation, with some scripts for native packaging.
Maintainers: currently I, @bpaquet, am the only maintainer. I will keep dependencies up to date, update the core to follow node version, but I do not have time to add features to the core. See Contributing below.
Weaknesses :
Which Pull Requests (PR) will be merged?
Conditions to have a PR merged :
Readme.md
.package.json
You are encouraged to ask to merge plugins without tests, which are not production ready.
After install, just add your config files to /etc/node-logstash/plugins.conf.d
, and restart node-logstash service node-logstash restart
.
To see what options are passed to node-logstash, see here.
To change log level, do node-logstash config:set LOG_LEVEL=debug
, and restart node-logstash.
apt-get install build-essential
yum install gcc gcc-c++ make
apt-get install libzmq1
. Under recent releases, this package is present in default repositories. On ubuntu lucid, use this ppa. On debian squeeze, use backports.yum install zeromq zeromq-devel
. Before, you have to add the rpm zeromq repo : curl http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/home:fengshuo:zeromq.repo > /etc/yum.repos.d/zeromq.repo
git clone git://github.com/bpaquet/node-logstash.git && cd node-logstash
npm install
.The executable is bin/node-logstash-agent
There are two formats for configuration:
Note: if you are using multiple config files, you can mix formats.
A plugin is instantiated by an URL. Example: input://file:///tmp/toto.log
. This URL
instantiates an input file plugin which monitors the file /tmp/toto.log
.
The URLs can be specified:
--config_file
switch)--config_dir
switch)Example for an input file
input {
file {
path => '/tmp/toto.log'
}
}
You can use if
to have an event dependent configuration. See here for details.
As for URLs, config can be specified
--config_file
switch)--config_dir
switch)Note: the implementation is young, all bugs reports are welcome. Note: both formats can be mixed.
--log_level
to change the log level (emergency, alert, critical, error, warning, notice, info, debug)--log_file
to redirect logs to a log file.--patterns_directories
to add some directories (separated by , ), for loading config for regex plugin and grok plugins. Grok patterns files must be located under a grok
subdirectory for each specified directory.--db_file
to specify the file to use a database for file inputs (see below)--http_max_sockets
to specify the max sockets of http.globalAgent.maxSockets. Default to 100.--alarm_file
to specify a file which will be created if node-logstash goes in alarm mode (see below).Config file for an agent:
input {
file {
path => "/var/log/nginx/access.log"
}
}
output {
zeromq {
address => ["tcp://log_server:5555"]
}
}
Config file for log server:
input {
zeromq {
address => ["tcp://0.0.0.0:5555"]
}
}
filter {
regex {
pattern => http_combined
}
}
output {
elasticsearch {
host => localhost
port => 9200
}
}
You can add easily add your plugins :
Manually :
/var/my_plugins/inputs
, /var/my_plugins/outputs
, /var/my_plugins/filters
NODE_PATH=/var/my_plugins:/node_logstash_path/lib
inputs
, outputs
or filters
directory. In the plugin code, you can reference base plugins with var base_filter = require('lib/base_filter');
With native packaging
The plugins must be deployed in /var/db/node-logstash/custom_plugins
. All subdirectories already exist. The NODE_PATH is already set.
Common concepts / parameters :
Common concepts / parameters :
Common concepts / parameters :
Copyright 2012 - 2014 Bertrand Paquet
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.