Open viditmaniyar opened 9 years ago
You could try Supervisor + one of https://github.com/Supervisor/initscripts. supervisord.conf.example
has some basic default settings that should work. You may have to adjust the command to your liking.
I have certainly read about that in the documentation. This is about running it as an independent service with an init script.
@iamrudra yeah you can!
easy example (/etc/init/elastalert.conf):
start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]
exec /path_to/elastalert --params_here
same for systemd (you can stablish a relationship, so elastalert will start after elasticsearch is up):
[Unit]
Description=Elastalert
After=elasticsearch.service
[Service]
Type=simple
User=xx
Group=xx
Restart=on-failure
ExecStart=/path_to/elastalert --params_here
[Install]
WantedBy=multi-user.target
hope this helps.
@iamrudra I made an init script on RHEL, here it is:
#!/bin/bash
# elastalert startup script for elastalert
# pidfile: /var/run/elastalert.pid
# chkconfig: 2345 99 01
NAME=elastalert
PIDFILE=/var/run/$NAME.pid
ELASTALERT_DIR=/opt/elastalert
ELASTALERT_USER=elastalert
CONFIG_FILE=$ELASTALERT_DIR/config.yaml
ELASTALERT=/usr/local/bin/$NAME
. /etc/rc.d/init.d/functions
case $1 in
start)
echo -n $"Starting $NAME: "
cd $ELASTALERT_DIR
daemon --user="$ELASTALERT_USER" --pidfile="$PIDFILE" "$ELASTALERT --config $CONFIG_FILE &"
RETVAL=$?
pid=`ps -ef | grep python | grep elastalert | awk '{print $2}'`
if [ -n "$pid" ]; then
echo $pid > "$PIDFILE"
fi
;;
stop)
echo -n $"Stopping $NAME: "
killproc -p "$PIDFILE" -d 10 "$ELASTALERT"
RETVAL="$?"
echo
[ $RETVAL = 0 ] && rm -f "$PIDFILE"
;;
*)
echo "Usage: /etc/init.d/elastalert {start|stop}" ;;
esac
exit 0
Here is how I got it running with systemctl
Create the service file
vi /lib/systemd/system/elastalert.service
Add this to the file
[Unit]
Description=elastalert
After=multi-user.target
[Service]
Type=simple
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/elastalert
[Install]
WantedBy=multi-user.target
Then create a link, reload the daemon, enable the service and start the service
ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service
systemctl daemon-reload
systemctl enable elastalert.service
systemctl start elastalert.service
systemctl status elastalert.service
FWIW, this is a slightly enhanced (arguably) version of an /etc/init.d/elastalert
script handling a few more common commands (restart, etc.) and capturing stderr/stdout and logging them:
#!/bin/bash
# elastalert startup script for elastalert
# pidfile: /var/run/elastalert.pid
# chkconfig: 2345 99 01
### BEGIN INIT INFO
# Provides: elastalert
# Required-Start: cgconfig
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: elastalert control
# Description:
### END INIT INFO
if [ $(id -u) -ne 0 ]; then
echo "This script can be run by root only. Exiting."
exit 4
fi
NAME=elastalert
PIDFILE=/var/run/$NAME.pid
LOCKFILE=/var/lock/subsys/$NAME
ELASTALERT_DIR=/opt/perf-dept/elastalert
ELASTALERT_USER=elastalert
CONFIG_FILE=$ELASTALERT_DIR/config.yaml
ELASTALERT=/usr/bin/$NAME
. /etc/rc.d/init.d/functions
[ -e /etc/sysconfig/$NAME ] && . /etc/sysconfig/$NAME
start() {
[ -x $ELASTALERT ] || exit 5
[ -f $CONFIG_FILE ] || exit 6
echo -n $"Starting $NAME: "
cd $ELASTALERT_DIR
daemon --user="$ELASTALERT_USER" --pidfile="$PIDFILE" "$ELASTALERT --config $CONFIG_FILE 2>&1 | /bin/logger -p daemon.info -t $NAME > /dev/null 2>&1 &"
retval=$?
pid=$(ps -ef | grep python | grep $NAME | awk '{print $2}')
if [ -n "$pid" ]; then
echo $pid > "$PIDFILE"
fi
echo
[ $retval -eq 0 ] && touch "$LOCKFILE"
return $retval
}
stop() {
echo -n $"Stopping $NAME: "
killproc -p "$PIDFILE" -d 10 "$NAME"
retval=$?
echo
[ $retval -eq 0 ] && rm -f "$LOCKFILE" "$PIDFILE"
return $retval
}
restart() {
stop
start
}
reload() {
restart
}
force_reload() {
restart
}
rh_status() {
# run checks to determine if the service is running or use generic status
status $NAME
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
exit 2
esac
exit $?
On centos 7 and tried using the script below with no luck:
#!/bin/bash
# myapp daemon
# chkconfig: 345 20 80
# description: myapp daemon
# processname: myapp
DAEMON_PATH="/elastalert/elastalert"
NAME=elastalert
DESC="elastalert"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
case "$1" in
start)
printf "%-50s" "Starting $NAME..."
cd $DAEMON_PATH
daemon --user="$ELASTALERT_USER" --pidfile="$PIDFILE" "$ELASTALERT --config $CONFIG_FILE 2>&1 | /bin/logger -p daemon.info -t $NAME > /dev/null 2>&1 &"
retval=$?
# PID=`$DAEMON > /dev/null 2>&1 & echo $!`
echo $PID
#echo "Saving PID" $PID " to " $PIDFILE
if [ -z $PID ]; then
printf "%s\n" "Fail"
else
echo $PID > $PIDFILE
printf "%s\n" "Ok"
fi
;;
status)
printf "%-50s" "Checking $NAME..."
if [ -f $PIDFILE ]; then
PID=`cat $PIDFILE`
if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
printf "%s\n" "Process dead but pidfile exists"
else
echo "Running"
fi
else
printf "%s\n" "Service not running"
fi
;;
stop)
printf "%-50s" "Stopping $NAME"
PID=`cat $PIDFILE`
cd $DAEMON_PATH
if [ -f $PIDFILE ]; then
kill -HUP $PID
printf "%s\n" "Ok"
rm -f $PIDFILE
else
printf "%s\n" "pidfile not found"
fi
;;
restart)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {status|start|stop|restart}"
exit 1
esac
I run elastalert from the command line like this /elastalert/elastalert/python -m elastalert --verbose --rule rules_folder/frequency.yaml --config config.yaml --config config.yaml
Thanks in advance!
Here's how I had to tweak the file to start on Debian Linux: (Thanks everyone for your suggestions!)
[Unit] Description=Elastalert After=network.target
[Service] Type=simple User=root Group=root WorkingDirectory=/opt/elastalert ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config /opt/elastalert/config.yaml StandardOutput=syslog StandardError=syslog KillSignal=SIGKILL PIDFile=/var/run/elastalert.pid
[Install] WantedBy=multi-user.target
@JimKeating2 @Qmando @elvarb
I followed the above steps to start elastalert as service but I get the below error message. Please assist.
● elastalert.service - Elastalert
Loaded: loaded (/lib/systemd/system/elastalert.service; linked; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-07-10 10:45:17 UTC; 2s ago
Process: 7933 ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config **/opt/elastalert/config.yaml (code=exited, status=1/FAILURE)
Main PID: 7933 (code=exited, status=1/FAILURE)**
Jul 10 10:45:17 infosec.novalocal python[7933]: File "/opt/elastalert/elastalert/elastalert.py", line 1775, in main
Jul 10 10:45:17 infosec.novalocal python[7933]: client = ElastAlerter(args)
Jul 10 10:45:17 infosec.novalocal python[7933]: File "/opt/elastalert/elastalert/elastalert.py", line 121, in __init__
Jul 10 10:45:17 infosec.novalocal python[7933]: self.conf = load_rules(self.args)
Jul 10 10:45:17 infosec.novalocal python[7933]: File "elastalert/config.py", line 453, in load_rules
**Jul 10 10:45:17 infosec.novalocal python[7933]: raise EAException('Error loading file %s: %s' % (rule_file, e))**
**Jul 10 10:45:17 infosec.novalocal python[7933]: elastalert.util.EAException: Error loading file example_rules/example_new_term.yaml: Error initializing r...**
Jul 10 10:45:17 infosec.novalocal systemd[1]: elastalert.service: main process exited, code=exited, status=1/FAILURE
Jul 10 10:45:17 infosec.novalocal systemd[1]: Unit elastalert.service entered failed state.
Jul 10 10:45:17 infosec.novalocal systemd[1]: elastalert.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
elastalert service
[Unit]
Description=Elastalert
After=network.target
[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config /opt/elastalert/config.yaml
StandardOutput=syslog
StandardError=syslog
KillSignal=SIGKILL
PIDFile=/var/run/elastalert.pid
[Install]
WantedBy=multi-user.target
Omando, I'd say first of all check the paths to make sure every path is correct. Also, the service will not start if one of the rules is not configured correctly. It looks like it is one of the rules that is stopping the service from running correctly. (Rule in the "example_rules" folder) Does it run from the command line correctly? I suggest running Elastalert from the command line to check your rules...especially at first.
This is a good command for testing your rules: root@ElastAlert:/opt/elastalert# elastalert-test-rule --count-only rule_testing/event_id_13.yaml
Best Wishes! JK
@sathishdsgithub
You are using the example rules, unmodified?
If you just run elastalert manually, you will see the full error
In your output, you can only see
Error loading file example_rules/example_new_term.yaml: Error initializing r...
I created a new folder for rules and configured Elastalert to only look in the new folder for any rules. I believe the location is set in the ElastAlert.yaml config file.
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: my_rules
The rules have to be just right or you will get an error every time. We created are using Winlogbeat on Elasticsearch for our Windows Server logs. Here's an example of a rule that looks for failed password changes:
# Alert when the rate of events exceeds a threshold
# (Optional)
# Elasticsearch host
es_host: host1.mydomain.org
# (Optional)
# Elasticsearch port
es_port: 9200
# (OptionaL) Connect with SSL to Elasticsearch
use_ssl: False
run_every:
seconds: 40
aggregation:
seconds: 40
realert:
seconds: 0
# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# (Required)
# Rule name, must be unique
name: Failed_Password_Change
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: any
# (Required)
# Index to search, wildcard supported
index: winlogbeat-*
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
# num_events: 1
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
# timeframe:
# minutes: 2
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:
event_id: "4723"
- term:
keywords: "Audit Failure"
# (Required)
# The alert is use when a match is found
alert:
- "email"
from_addr: "elastalert@mydomain.org"
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "isalerts@mydomain.org"
Best Wishes...JK
@JimKeating2 @Qmando I fix the error by modifying the correct folder path. However, I'm unable to make the service to run during startup. I get failed to execute the operation. Please assist.
[root@infosec ~]# systemctl enable elastalert.service
**Failed to execute operation: Too many levels of symbolic links**
I followed the below procedure
more /etc/systemd/system/elastalert.service
[Unit]
Description=Elastalert
After=network.target
[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/opt/elastalert
ExecStart=/usr/bin/python -m elastalert.elastalert --verbose --config /opt/elastalert/config.yaml
StandardOutput=syslog
StandardError=syslog
KillSignal=SIGKILL
PIDFile=/var/run/elastalert.pid
[Install]
WantedBy=multi-user.target
`ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service`
I would remove the service as it looks as if you have multiple service links:
Here's a good link to show you how: https://superuser.com/questions/513159/how-to-remove-systemd-services (Find and remove all files: find / -name elastalert.service)
Recreate service file in /lib/systemd/system/elastalert.service
Create Link: ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service
Reload Daemon: systemctl daemon-reload
Enable Service and Start Service: systemctl enable elastalert.service systemctl start elastalert.service systemctl status elastalert.service
I found zdaemon 4.2.0 to be a pretty good solution and easy.
pip install zdaemon
Create a config file with (ie. zdaemon_conf):
<runner>
program python -m elastalert.elastalert --conf /path_to_config/config.yaml
socket-name /tmp/elastalert.zdsock
forever true
</runner>
then run:
zdaemon -C /path_to_config/zdaemon_conf start
@flippakitten
Let me try and update you. Can we specify the zdaemon config file in /etc path ?
@sathishdsgithub Not totally sure I understand what you mean but you can keep the zdaemon config file anywhere as long as you have permission to it. (sorry it's late)
@flippakitten
How do I enable the zdaemon to start during boot ?
pip install zdaemon
<runner>
program python -m elastalert.elastalert --conf /path_to_config/config.yaml
socket-name /tmp/elastalert.zdsock
forever true
</runner>
zdaemon -C /path_to_config/zdaemon_conf start
@sathishdsgithub I haven't implemented it to start at boot.
It would depend on the use case but I guess I would create a simple bash script and then call it from a crontab on reboot. This comes with other possibly issues but if the goal is to get it working for now, it should do the trick.
Something like:
create a file ie. 'start_elastalert.s'h and add
#!/bin/sh
zdaemon -C /path_to_config/zdaemon_conf start
Then
$ chmod +x start_elastalert.sh
$ crontab -e
#add cron
@reboot /path/to/script/start_elastalert.sh
Is there a way we can add an init script for ElastAlert and run it as a service? This would be a nice enhancement to have.