slackclient module is deprecated

JoshuaSmeda / thehive_sla_monitor

Query and cross-check TheHive (SIRP) alerts based on set severity statuses, and automatically perform various escalations based on your configuration. Integrates with Slack, Twilio, Flask and TheHive.

https://thehive-project.org/

GNU General Public License v3.0

9 stars 1 forks source link

slackclient module is deprecated #1

Closed devatnull closed 4 years ago

devatnull commented 4 years ago

Hello Joshua,

I think the slackclient module is deprecated, there is a module named "slack" but, it still wont work with the current script.

$ python3 bot.py 
Traceback (most recent call last):
  File "bot.py", line 13, in <module>
    from slackclient import SlackClient
ModuleNotFoundError: No module named 'slackclient'

JoshuaSmeda commented 4 years ago

Hello @devatnull,

Thanks, I will fix and merge.

devatnull commented 4 years ago

I thank you, greetins to Nclose from Turkey.

devatnull commented 4 years ago

Hello Joshua,

I removed slack and call functionality completely,

Traceback (most recent call last):
  File "bot.py", line 16, in <module>
    from soc_bot.logger import collect_logs, log_current_time
  File "/home/perkin/twilioTest/thehive_slack_bot/soc_bot/logger.py", line 2, in <module>
    import dateutil.parser
ModuleNotFoundError: No module named 'dateutil'

after that I removed the logger functionality.

* Serving Flask app "bot" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:3001/ (Press CTRL+C to quit)

127.0.0.1 - - [24/Jul/2020 11:00:24] "GET / HTTP/1.1" 404 -

curl 127.0.0.1:3001
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>

There is an alert over 1 hour on the hive, but the server still doesn't send any SMS, I tested my twilio SMS funcionality with basic framework, it works fine.

I am not very good with Python as a sysadmin, having hard times and no luck, Here is the edited code:

from __future__ import print_function
from __future__ import unicode_literals

import time as t
import sys
import json, ast
import threading
import os
import re
from multiprocessing import Process, Queue, Manager
from flask import Flask, request, make_response, redirect
from datetime import datetime, time
#from slackclient import SlackClient
from twilio.rest import Client
from thehive4py.api import TheHiveApi
from thehive4py.query import *
#from soc_bot.templates import slack_bot_pick_user_template, slack_bot_alert_notice_template, slack_bot_alert_notice_update, slack_bot_alert_notice_ignore
#from soc_bot.logger import collect_logs, log_current_time

# Instantiate Hive variables
sla_30 = 1800
sla_45 = 2700
sla_60 = 3600
hive_30_list = []
hive_30_dict = {}
hive_45_list = []
hive_45_dict = {}
hive_60_list = []
hive_60_dict = {}
ignore_list = []
called_list = []
hive_api_key = '*'

# Instantiate Slack & Twilio variables
twilio_number = "*"
twilio_auth_token = '*'
twilio_account_sid = '*'

# Initialize API classes
twilio_client = Client(twilio_account_sid, twilio_auth_token)
api = TheHiveApi('http://192.168.122.30:9000', hive_api_key)

# Instatiate Flask classes
app = Flask(__name__)
manager = Manager()
alert_dict = manager.dict()

def create_hive30_dict(id, rule_name, alert_date, alert_age):
  print("Updating 30M SLA dictionary")
  hive_30_dict.update({ id : rule_name, })
  send_sms()

def create_hive45_dict(id, rule_name, alert_date, alert_age):
  print("Updating 45M SLA dictionary")
  hive_45_dict.update({ id : rule_name, })
  make_call(id)

def create_hive60_dict(id, rule_name, alert_date, alert_age):
  hive_60_dict.update({ id : rule_name, })
  collect_logs(str(log_current_time()) + ' Alert ID: ' + id + ' Rule Name: ' + rule_name + ' Alert Date: ' + alert_date + ' Alert Age: ' + alert_age)
  # escalate to a senior

def add_to_30m(id):
  if id in hive_30_list:
    print('Already added - 30 minute SLA list: ' + id)
  else:
    print('Appending - 30 minute SLA list: ' + id)
    hive_30_list.append(id)

def add_to_45m(id):
  if id in hive_45_list:
    print('Already added - 45 minute SLA list: ' + id)
  else:
    print('Appending - 45 minute SLA list: ' + id)
    hive_45_list.append(id)

def add_to_60m(id):
  if id in hive_60_list:
    print('Already added - 60 minute SLA list: ' + id)
  else:
    print('Appending - 60 minute SLA list: ' + id)
    hive_60_list.append(id)

def clean_ignore_list(id):
  time.sleep(3600)
  print("Removing " + id + " from ignore list")
  ignore_list.remove(id)

def add_to_temp_ignore(id):
  print("Adding " + id + " to ignore list")
  ignore_list.append(id)
  ignore_thread = threading.Thread(target=clean_ignore_list)
  ignore_thread.start()
  return id

def add_to_called_list(id):
  print("Adding " + id + " to called list")
  called_list.append(id)
  return id

def search(title, query):
  current_date = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
  current_date = datetime.strptime(current_date, '%Y-%m-%d %H:%M:%S')
  response = api.find_alerts(query=query)

  if response.status_code == 200:
    data = json.dumps(response.json())
    jdata = json.loads(data)
    for d in jdata:
      if d['severity'] == 3:
        ts = int(d['createdAt'])
        ts /= 1000
        alert_date = datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
        print(d['id'] + " " + d['title'] + " " + str(alert_date))
        alert_date = datetime.strptime(alert_date, '%Y-%m-%d %H:%M:%S')
        diff = (current_date - alert_date)
        if sla_30 < diff.total_seconds() and sla_45 > diff.total_seconds():
          print("30/M SLA: " + str(diff) + " " + d['id'])
          create_hive30_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_30m(d['id'])
        elif sla_45 < diff.total_seconds() and sla_60 > diff.total_seconds():
          print("45/M SLA: " + str(diff) + " " + d['id'])
          create_hive45_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_45m(d['id'])
        elif sla_60 < diff.total_seconds():
          print("60/M SLA: " + str(diff) + " " + d['id'])
          create_hive60_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_60m(d['id'])

    print('')

  else:
    print('ko: {}/{}'.format(response.status_code, response.text))

def send_sms():
  for k,v in hive_30_dict.iteritems():
    if k in hive_30_list:
      print("Already sent SMS regarding ID: " + k)
    else:
      body = 'Hive Alert SLA Notice - ' + "\n" + v
      send_from = twilio_number
      send_to = '*'
      message = twilio_client.messages.create(body=body, from_=send_from, to=send_to)
      print("SMS Sent: " + str(message.sid))
      try:
        add_to_30m(k)
      except Exception as error:
        print("Error when adding to list after sending SMS: " + str(error))

# Print call list for record purposes?
#    calls = twilio_client.calls.list(limit=1)
#    for record in calls:
#      print(str(record.start_time) + " " + str(record.to) + " " + str(record.status))
#   https://www.twilio.com/docs/voice/api/call#parameters

def promote_to_case(case_id):
  print("Promoting Alert " + case_id)
  response = api.promote_alert_to_case(case_id)
  if response.status_code == 201:
    data = json.dumps(response.json())
    jdata = json.loads(data)
    case_id = (jdata['id'])
    return case_id
  else:
    print('ko: {}/{}'.format(response.status_code, response.text))

def bot_start():
  while True:
    search('Formatted DATA:', Eq('status', 'New'))
    t.sleep(120)

def webserver_start():
    app.run(port=3000, host='0.0.0.0')
    return

if __name__ == '__main__':
  webserver_start = Process(target=webserver_start)
  bot_start = Process(target=bot_start)
  bot_start.start()
  webserver_start.start()
  bot_start.join()
  webserver_start.join()

JoshuaSmeda commented 4 years ago

To resolve the Slack dependency issue. It's abit tricky since I'm not using the latest WebClient version!

:/tmp# python3 -m venv venv

:/tmp# . venv/bin/activate
(venv) :/tmp# ls
agent.txt  bot.py  logs  soc_alerter  soc_bot.log  standby.bak  templates  test.py  tests  venv
(venv) root@soc1:/opt/soc_alerter# pip freeze
pkg-resources==0.0.0

(venv) :/tmp# python
Python 3.5.2 (default, Jul 17 2020, 14:04:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import slack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'slack'

>>> import slackclient
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'slackclient'
>>> quit()

(venv) :/tmp# python -m pip install slackclient
WARNING: The directory '/home/joshua/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting slackclient
  Downloading slackclient-1.3.2.tar.gz (16 kB)
Collecting websocket-client<0.55.0,>=0.35
  Downloading websocket_client-0.54.0-py2.py3-none-any.whl (200 kB)
     |████████████████████████████████| 200 kB 10.2 MB/s 
Collecting requests<3.0a0,>=2.11
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 9.8 MB/s 
Collecting six<2.0a0,>=1.10
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 9.4 MB/s 
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 12.0 MB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
     |████████████████████████████████| 156 kB 10.3 MB/s 
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 11.1 MB/s 
Using legacy setup.py install for slackclient, since package 'wheel' is not installed.
Installing collected packages: six, websocket-client, chardet, idna, certifi, urllib3, requests, slackclient
    Running setup.py install for slackclient ... done
Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 six-1.15.0 slackclient-1.3.2 urllib3-1.25.10 websocket-client-0.54.0

(venv) :/tmp# python
Python 3.5.2 (default, Jul 17 2020, 14:04:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import slackclient
>>>

>>> from slackclient import SlackClient
>>>

Packages that I have installed - pip freeze:

certifi==2020.6.20
chardet==3.0.4
idna==2.10
pkg-resources==0.0.0
requests==2.24.0
six==1.15.0
slackclient==1.3.2
urllib3==1.25.10
websocket-client==0.54.0

JoshuaSmeda commented 4 years ago

Did you fill in your Twilio details?

# Instantiate Slack & Twilio variables
twilio_number = "*"
twilio_auth_token = '*'
twilio_account_sid = '*'

You also need to fill in the number you wish to send to. You can refer to this guide to find out the number format - https://www.twilio.com/docs/sms/send-messages

def send_sms():
  for k,v in hive_30_dict.iteritems():
    if k in hive_30_list:
      print("Already sent SMS regarding ID: " + k)
    else:
      body = 'Hive Alert SLA Notice - ' + "\n" + v
      send_from = twilio_number
      send_to = '*'
      message = twilio_client.messages.create(body=body, from_=send_from, to=send_to)
      print("SMS Sent: " + str(message.sid))
      try:
        add_to_30m(k)
      except Exception as error:
        print("Error when adding to list after sending SMS: " + str(error))

If you still don't come right, a error message should be logged if it doesn't send. You can paste it here so I can assist you better!

devatnull commented 4 years ago

Did you fill in your Twilio details?

# Instantiate Slack & Twilio variables
twilio_number = "*"
twilio_auth_token = '*'
twilio_account_sid = '*'

You also need to fill in the number you wish to send to. You can refer to this guide to find out the number format - https://www.twilio.com/docs/sms/send-messages

def send_sms():
  for k,v in hive_30_dict.iteritems():
    if k in hive_30_list:
      print("Already sent SMS regarding ID: " + k)
    else:
      body = 'Hive Alert SLA Notice - ' + "\n" + v
      send_from = twilio_number
      send_to = '*'
      message = twilio_client.messages.create(body=body, from_=send_from, to=send_to)
      print("SMS Sent: " + str(message.sid))
      try:
        add_to_30m(k)
      except Exception as error:
        print("Error when adding to list after sending SMS: " + str(error))

Yep, I filled in all the details needed but removed them while posting here, since they are sensitive information, will check and reply to you in a bit.

devatnull commented 4 years ago

Thanks for your help Joshua, I installed the same modules as you did;

Traceback (most recent call last):
  File "bot.py", line 17, in <module>
    from soc_bot.templates import slack_bot_pick_user_template, slack_bot_alert_notice_template, slack_bot_alert_notice_update, slack_bot_alert_notice_ignore
ImportError: cannot import name 'slack_bot_pick_user_template'

Traceback (most recent call last):
  File "bot.py", line 18, in <module>
    from soc_bot.logger import collect_logs, log_current_time
  File "/home/perkin/twilioTest/new/thehive_slack_bot/soc_bot/logger.py", line 2, in <module>
    import dateutil.parser
ModuleNotFoundError: No module named 'dateutil'

then removed template and the logger, still no luck.

Is there a way to parse alerts from TheHive to a Twilio API just for SMS? I was checking Twilio docs and Hive4py,

from twilio.rest import Client
account_sid = "*"
auth_token = "*"
client = Client(account_sid, auth_token)
client.messages.create(
  to="*",
  from_="*",
  body="Bu mesaj sana ulasirsa bana geri donus yap.")

And maybe using this with hive4py to receive alerts and then to send SMS would work perfectly for me, but, 2020-07-24-140005_289x293_scrot

Samples from hive4py doesn't contain something like "GET ALERTS" been researching for 2 days and still no luck, if may I ask, could you point me in any direction in order for me to code something that would do the following; 1- Receive hive alerts (Instant or 10 minutes, doesn't matter) 2- Send the alert via Twilio API

Sorry for bothering you this much Joshua, I really appreciate it.

devatnull commented 4 years ago

facepalm curl -X GET -H 'Authorization: Bearer ' https:///api/alert/ec253f004d406bd7afaa4d8a99212538

JoshuaSmeda commented 4 years ago

Hi,

To answer your questions:

1- Receive hive alerts (Instant or 10 minutes, doesn't matter) 2- Send the alert via Twilio API

Hive alerts are queried every 2 minutes by this function:

def bot_start():
  while True:
    search('Formatted DATA:', Eq('status', 'New'))
    t.sleep(120)

If you want to alert on 10 minutes, adjust the search function to include 10 minute actions:

You will need to create a 10 minute second variable like such:

sla_30 = 1800
sla_45 = 2700
sla_60 = 3600

Then add 10 minutes to the below before the 30 minute check:

def search(title, query):
  current_date = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
  current_date = datetime.strptime(current_date, '%Y-%m-%d %H:%M:%S')
  response = api.find_alerts(query=query)

  if response.status_code == 200:
    data = json.dumps(response.json())
    jdata = json.loads(data)
    for d in jdata:
      if d['severity'] == 3:
        ts = int(d['createdAt'])
        ts /= 1000
        alert_date = datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
        print(d['id'] + " " + d['title'] + " " + str(alert_date))
        alert_date = datetime.strptime(alert_date, '%Y-%m-%d %H:%M:%S')
        diff = (current_date - alert_date)
        if sla_30 < diff.total_seconds() and sla_45 > diff.total_seconds():
          print("30/M SLA: " + str(diff) + " " + d['id'])
          create_hive30_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_30m(d['id'])
        elif sla_45 < diff.total_seconds() and sla_60 > diff.total_seconds():
          print("45/M SLA: " + str(diff) + " " + d['id'])
          create_hive45_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_45m(d['id'])
        elif sla_60 < diff.total_seconds():
          print("60/M SLA: " + str(diff) + " " + d['id'])
          create_hive60_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_60m(d['id'])

    print('')

Then add a 10 minute SLA list function to track the ID so it doesn't spam alert you like such:

def add_to_30m(id):
  if id in hive_30_list:
    print('Already added - 30 minute SLA list: ' + id)
  else:
    print('Appending - 30 minute SLA list: ' + id)
    hive_30_list.append(id)

And lastly a 10 minute dictionary list to store the alert data for future use (via sms):

def create_hive30_dict(id, rule_name, alert_date, alert_age):
  print("Updating 30M SLA dictionary")
  hive_30_dict.update({ id : rule_name, })
  slack_bot_notice_alert(channel, id, rule_name, alert_date, alert_age)
  send_sms()

You can use the existing code and just copy and paste based on your need. No extra functionality is necessary as what you're trying to achieve, has already been done.

Hope this helps, let me know if you get stuck!

devatnull commented 4 years ago

hey Joshua,

had to change the code a bit like you said, removed deprecated functions like "iteritems" it works in python2 not in python3, I have a question if you dont mind. How can I change what message is going to be sent? the body of the message is currently only the name of the alert. But I want to add lets say, source IP, node, data, and artifacts. I don't know maybe in your original code it was like that and its now working like mentioned above because I changed ".iteritems" ".items"

    {
        "_id": "1465afc8a8c70ce27ee53c2433cfd98e",
        "_parent": null,
        "_routing": "1465afc8a8c70ce27ee53c2433cfd98e",
        "_type": "alert",
        "_version": 2,
        "artifacts": [
            {
                "data": "192.168.122.155",
                "dataType": "alerting_ip",
                "ioc": false,
                "message": null,
                "sighted": false,
                "tags": [],
                "tlp": 2
            }
            ],
        "case": "AXOPhKTRryEHUBgvN3yj",
        "createdAt": 1595836942511,
        "createdBy": "schive",
        "customFields": {},
        "date": 1595836942512,
        "description": "N/A",
        "follow": true,
        "id": "1465afc8a8c70ce27ee53c2433cfd98e",
        "lastSyncDate": 1595836942512,
        "severity": 2,
        "source": "instance1",
        "sourceRef": "alerat-ref",
        "status": "Imported",
        "title": "New Alert",
        "tlp": 2,
        "type": "external",
        "updatedAt": 1595840702678,
    }

JoshuaSmeda commented 4 years ago

Hi devatnull,

I'll implement the 10 minute alert threshold and a feature for you to select what information you want to send to sms tomorrow.

I'll also fix the dependency issues you have so it will hopefully work smoothly out of the box for you.

Sorry for all the hassles you've been having with this project, will get it sorted soon.

On Tue, 28 Jul 2020, 14:43 devatnull, notifications@github.com wrote:

hey Joshua,

had to change the code a bit like you said, removed deprecated functions like "iteritems" it works in python2 not in python3, I have a question if you dont mind. How can I change what message is going to be sent? the body of the message is currently only the name of the alert. But I want to add lets say, source IP, node, data, and artifacts. I don't know maybe in your original code it was like that and its now working like mentioned above because I changed ".iteritems" ".items"
{
    "_id": "1465afc8a8c70ce27ee53c2433cfd98e",
    "_parent": null,
    "_routing": "1465afc8a8c70ce27ee53c2433cfd98e",
    "_type": "alert",
    "_version": 2,
    "artifacts": [
        {
            "data": "192.168.122.155",
            "dataType": "alerting_ip",
            "ioc": false,
            "message": null,
            "sighted": false,
            "tags": [],
            "tlp": 2
        }
        ],
    "case": "AXOPhKTRryEHUBgvN3yj",
    "createdAt": 1595836942511,
    "createdBy": "schive",
    "customFields": {},
    "date": 1595836942512,
    "description": "N/A",
    "follow": true,
    "id": "1465afc8a8c70ce27ee53c2433cfd98e",
    "lastSyncDate": 1595836942512,
    "severity": 2,
    "source": "instance1",
    "sourceRef": "alerat-ref",
    "status": "Imported",
    "title": "New Alert",
    "tlp": 2,
    "type": "external",
    "updatedAt": 1595840702678,
}
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/JoshuaSmeda/thehive_slack_bot/issues/1#issuecomment-665016919, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJRBWDXZCIVNFQNDQKQDIUDR53BYHANCNFSM4PF3YJBA .

devatnull commented 4 years ago

Thank you Joshua, I wouldn't want to cost you your dear time.

I've changed the code according to my needs, its now 100 lines long, sends sms every 10 minutes, removed slack and call function to keep it clean first, it is working perfectly. Your code is very cleverly written I can say that.

def send_sms():
  for k,v in hive_10_**dict.items()**:
    if k in hive_10_list:
      print("Already sent SMS regarding ID: " + k)
    else:
      body = 'Hive Alert Notice - ' + "\n" + "Alert Title:  " + v + "\n" + "Alert ID: " + k + "\n" + **search(query['artifacts'])**
      send_from = twilio_number
      send_to = '$SMS_TO'
      message = twilio_client.messages.create(body=body, from_=send_from, to=send_to)
      print("SMS Sent: " + str(message.sid))
      try:
        add_to_10m(k)
      except Exception as error:
        print("Error when adding to list after sending SMS: " + str(error))

I have turned what I want to show in the code to bold. First bold is, it was "dict.iteritems" had to make it "items" to work with python3, and the second bold part is the part where I want to add other data that is in the alert. I am quoting it down below. I've checked how to do this, still trying, I believe I will find a way, but if you can hit me with any tips, Iw ould be very glad.

Edit: apperantly you can't bold a text while it is inside quotes. but you will notice search(query['artifacts']) on the body part. my implementation of this is totaly wrong since search is something else defined. and query is in that. I will probably add external variables and try.

Thank you for your kindness.

        "artifacts": [
            {
                "data": "192.168.122.155",
                "dataType": "alerting_ip",
                "ioc": false,
                "message": null,
                "sighted": false,
                "tags": [],
                "tlp": 2
            }
            ],

Currently it is only the alert name. and the id. If i can find a way to parse any part of the alert to the body of the SMS, I am gonna be rock solid :) WhatsApp Image 2020-07-28 at 11 17 36

JoshuaSmeda commented 4 years ago

Hi @devatnull,

I have a solution for you. Will send it through today still.

devatnull commented 4 years ago

Thanks a lot Joshua, I am currently working on graylog clusters, noticed that I didn't send the code, sorry for that. SMS script is currently something like this;

from __future__ import print_function
from __future__ import unicode_literals

import time as t
import sys
import json, ast
import threading
import os
import re
from multiprocessing import Process, Queue, Manager
from flask import Flask, request, make_response, redirect
from datetime import datetime, time
from slackclient import SlackClient
from twilio.rest import Client
from thehive4py.api import TheHiveApi
from thehive4py.query import *
from log_bot.logger import collect_logs, log_current_time
from log_bot.search import search

sla_10 = 600
sla_30 = 1800
hive_10_list = []
hive_10_dict = {}

hive_api_key = '$HIVE_API'

# Instantiate Slack & Twilio variables
twilio_number = "$TWILIO_NUMBER"
twilio_auth_token = '$TWILIO_AUTH_TOKEN'
twilio_account_sid = '$TWILIO_ACCOUND_SID'

# Initialize API classes
twilio_client = Client(twilio_account_sid, twilio_auth_token)
api = TheHiveApi('http://192.168.122.30:9000', hive_api_key)

# Instatiate Flask classes
app = Flask(__name__)
manager = Manager()
alert_dict = manager.dict()

# Initialize series for data extraction for SMS
#artifactResponse = api.find_alerts(query)
#artifactData = json.dumps(artifactResponse.json())
#jsonArtifact = json.loads(artifactData)

def create_hive10_dict(id, rule_name, alert_date, alert_age):
  print("Updating 10M SLA dictionary")
  hive_10_dict.update({ id : rule_name})
  send_sms()
#  collect_logs(str(log_current_time()) + ' Alert ID: ' + id + ' Rule Name: ' + rule_name + ' Alert Date: ' + alert_date + ' Alert Age: ' + alert_age)

def add_to_10m(id):
  if id in hive_10_list:
    print('Already added - 10 minute SLA list: ' + id)
  else:
    print('Appending - 10 minute SLA list: ' + id)
    hive_10_list.append(id)

def search(title, query):
  current_date = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
  current_date = datetime.strptime(current_date, '%Y-%m-%d %H:%M:%S')
  response = api.find_alerts(query=query)

  if response.status_code == 200:
    data = json.dumps(response.json())
    jdata = json.loads(data)
    for d in jdata:
      if d['severity'] > 2:
        #hive_10_list.append(['description'])
        ts = int(d['createdAt'])
        ts /= 1000
        alert_date = datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
        print(d['id'] + " " + d['title'] + " " + d['description']  + " " + str(alert_date))
        alert_date = datetime.strptime(alert_date, '%Y-%m-%d %H:%M:%S')
        diff = (current_date - alert_date)
        if sla_10 < diff.total_seconds() and sla_30 > diff.total_seconds():
          print("10/M SLA: " + str(diff) + " " + d['id'])
          create_hive10_dict(d['id'], d['title'], str(alert_date), str(diff))
          add_to_10m(d['id'])
    print('︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵ ︵‿︵')

def send_sms():
  for k,v in hive_10_dict.items():
    if k in hive_10_list:
      print("Already sent SMS regarding ID: " + k)
    else:
      body = 'Hive Alert Notice - ' + "\n" + "Alert Title:  " + v + "\n" + "Alert ID: " + k + "\n"
      send_from = twilio_number
      send_to = '$NUMBER'
      message = twilio_client.messages.create(body=body, from_=send_from, to=send_to)
      print("SMS Sent: " + str(message.sid))
      try:
        add_to_10m(k)
      except Exception as error:
        print("Error when adding to list after sending SMS: " + str(error))

def bot_start():
  while True:
    search('Formatted DATA:', Eq('status', 'New'))
    t.sleep(120)

def webserver_start():
    app.run(port=3000, host='192.168.122.30')
    return

if __name__ == '__main__':
  webserver_start = Process(target=webserver_start)
  bot_start = Process(target=bot_start)
  bot_start.start()
  webserver_start.start()
  bot_start.join()
  webserver_start.join()

JoshuaSmeda commented 4 years ago

Hi @devatnull ,

Fix has been pushed - https://github.com/JoshuaSmeda/thehive_slack_bot/tree/python-3-port/%231

So, you need to add all your information in configuration.py.

It will be injected into the application at run-time.

If a alert fires, you will notice I'm sending the d as well which is the TheHive response:

        if sla_30 < diff.total_seconds() and sla_45 > diff.total_seconds():
          logging.info("30/M SLA: " + str(diff) + " " + d['id'])
          create_hive30_dict(d['id'], d['title'], str(alert_date), str(diff), d)
          add_to_30m(d['id'])

It gets sent to the send_sms function send_sms(*args):

def create_hive30_dict(id, rule_name, alert_date, alert_age, *args):
  logging.info("Updating 30M SLA dictionary")
  hive_30_dict.update({ id : rule_name, })
  slack_bot_notice_alert(channel, id, rule_name, alert_date, alert_age)
  send_sms(*args)

It's here where you can choose what you want to collect:

Uncomment: # logging.info(alert_dump) # If you want to see all items in Hive response

And then choose what you want to "collect" and send:

          severity = (alert_json['severity']) # I want severity
          all_artifacts = (alert_json['artifacts']) # I want all artifacts
          data_artifacts = (alert_json['artifacts'][0]['data']) # I want only the data item which lies under the artifacts element

          msg_body = severity # Send only the severity
       #   msg_body = severity + " " + data_artifacts # Send severity and data_artifacts

def send_sms(*args):
  for alert_id, alert_name in hive_30_dict.items():
    if alert_id in hive_30_list:
      logging.warning("Already sent SMS regarding ID: " + alert_id)
    else:
      if not is_empty(args):
        for alert_data in args:
          alert_dump = json.dumps(alert_data, indent=2)
        #  logging.info(alert_dump) # If you want to see all items in Hive response
          alert_json = json.loads(alert_dump)

          severity = (alert_json['severity']) # I want severity
          all_artifacts = (alert_json['artifacts']) # I want all artifacts
          data_artifacts = (alert_json['artifacts'][0]['data']) # I want only the data item which lies under the artifacts element

          msg_body = severity # Send only the severity
       #   msg_body = severity + " " + data_artifacts # Send severity and data_artifacts

      else:
        msg_body = 'Please attend immediately!'

      twilio_msg_body = 'TheHive SLA Alert - %s \n %s' % (alert_id, msg_body)
      message = twilio_client.messages.create(body=twilio_msg_body, from_=configuration.TWILIO_SETTINGS['TWILIO_SENDER'], to=configuration.TWILIO_SETTINGS['TWILIO_RTCP'])
      t.sleep(2)
      logging.info('SMS Sent: ' + str(message.sid))
      try:
        add_to_30m(alert_id)
      except Exception as re:
        logging.error("Error when adding to list after sending SMS: " + str(re))

To pull and test this repo, you can do the following:

git clone repo_url -b python-3-port/#1

I hope this helps :-)

devatnull commented 4 years ago

Hey Joshua, you are amazing thank you. I've been testing and playing with the code since you committed it, I can add anything from an alert to the twilio SMS that I want, that is working perfect perfect.

But, I noticed that if some paramater that is defined in the SMS section, is not present in the alert itself, it will not be overseen but rather it will give an error. Lets say you have an alert like the following:

{
    "_id": "e6d1f7b1ede39ffce52999b428212dff",
    "_parent": null,
    "_routing": "e6d1f7b1ede39ffce52999b428212dff",
    "_type": "alert",
    "_version": 1,
    "artifacts": [
        {
            "data": "137.0.0.7",
            "dataType": "alerting_ip",
            "ioc": false,
            "message": null,
            "sighted": false,
            "tags": [],
            "tlp": 2
        },
        {
            "data": "HIGH_CPU",
            "dataType": "alerting_problem",
            "ioc": false,
            "message": null,
            "sighted": false,
            "tags": [],
            "tlp": 2
        }
    ],
    "caseTemplate": null,
    "createdAt": 1596053310733,
    "createdBy": "schive",
    "customFields": {
        "booleanField": {
            "boolean": true,
            "order": 0
        },
        "businessImpact": {
            "order": 1,
            "string": "HIGH"
        },
        "cvss": {
            "number": 9,
            "order": 3
        },
        "occurDate": {
            "date": 1596053310000,
            "order": 2
        }
    },
    "date": 1596053310000,
    "description": "N/A",
    "follow": true,
    "id": "e6d1f7b1ede39ffce52999b428212dff",
    "lastSyncDate": 1596053310734,
    "severity": 4,
    "source": "instance1",
    "sourceRef": "dc7452",
    "status": "New",
    "tags": [
        "TheHive4Py",
        "sample"
    ],
    "title": "THIS IS IT AND IT WILL SURELY WORK",
    "tlp": 3,
    "type": "external"
}

And the "def send_sms" is something like this:

def send_sms(*args):
  for alert_id, alert_name in hive_10_dict.items():
    if alert_id in hive_10_list:
      logging.warning("Already sent SMS regarding ID: " + alert_id)
    else:
      if not is_empty(args):
        for alert_data in args:
          alert_dump = json.dumps(alert_data, indent=2)
         # logging.info(alert_dump) # If you want to see all items in Hive response
          alert_json = json.loads(alert_dump)

          severity = (alert_json['severity']) # I want severity
          tags = (alert_json['tags'][0])
          tags_two = (alert_json['tags'][1])
          sourceRef = (alert_json['sourceRef'])
          source = (alert_json['source'])
          title = (alert_json['title'])
          description = (alert_json['description'])
          all_artifacts = (alert_json['artifacts']) # I want all artifacts
          data_two_artifacts = (alert_json['artifacts'][1]['data'])
          dataType_two_artifacts = (alert_json['artifacts'][1]['dataType'])
          data_artifacts = (alert_json['artifacts'][0]['data']) # I want only the data item which lies under the artifacts element
          dataType_artifacts = (alert_json['artifacts'][0]['dataType'])

          msg_body = "Title: " + title + " \n " + "Description: " + description + " \n " + "ID: " + alert_id + " \n " + "Source: " + source + " \n " + "SourceRef: " + sourceRef + " \n " + dataType_artifacts + ": " + data_artifacts  + " \n " + dataType_two_artifacts + ": " + data_two_artifacts + " \n " "Tag1: " + tags + " \n " + tags_two

      else:
        msg_body = 'Please attend immediately!'

      twilio_msg_body = 'Critical Alert Appeared on TheHive - \n ' +  " " + msg_body
      message = twilio_client.messages.create(body=twilio_msg_body, from_=twilio_number, to=twilio_recipient)
      t.sleep(2)
      logging.info('SMS Sent: ' + str(message.sid))
      try:
        add_to_10m(alert_id)
      except Exception as re:
        logging.error("Error when adding to list after sending SMS: " + str(re))

The message will be sent, because the alert that is parsed includes every parameter that is defined in the "msg_body" and "twilio_msg_body". But, lets say that there is an alert like this;

{
    "_id": "f21efac97ba17c3a1a791496c2b5946a",
    "_parent": null,
    "_routing": "f21efac97ba17c3a1a791496c2b5946a",
    "_type": "alert",
    "_version": 1,
    "artifacts": [
        {
            "data": "137.0.0.7",
            "dataType": "alerting_ip",
            "ioc": false,
            "message": null,
            "sighted": false,
            "tags": [],
            "tlp": 2
        },
        {
            "data": "HIGH_CPU",
            "dataType": "alerting_problem",
            "ioc": false,
            "message": null,
            "sighted": false,
            "tags": [],
            "tlp": 2
        }
    ],
    "caseTemplate": null,
    "createdAt": 1596053455968,
    "createdBy": "schive",
    "customFields": {
        "booleanField": {
            "boolean": true,
            "order": 0
        },
        "businessImpact": {
            "order": 1,
            "string": "HIGH"
        },
        "cvss": {
            "number": 9,
            "order": 3
        },
        "occurDate": {
            "date": 1596053455000,
            "order": 2
        }
    },
    "date": 1596053455000,
    "description": "N/A",
    "follow": true,
    "id": "f21efac97ba17c3a1a791496c2b5946a",
    "lastSyncDate": 1596053455968,
    "severity": 4,
    "source": "instance1",
    "sourceRef": "ccaae8",
    "status": "New",
    "tags": [],
    "title": "THIS IS IT AND IT WILL SURELY WORK",
    "tlp": 3,
    "type": "external"
}

Everything is the same except for the "tags" parameter, that is empty. Since the msg_body and twilio_msg_body has "Tag1: " + tags + " \n " + tags_two in the body and the alert does not have tags defined the script will give an error,

Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "bot.py", line 146, in bot_start
    search('Formatted DATA:', Eq('status', 'New'))
  File "bot.py", line 94, in search
    create_hive10_dict(d['id'], d['title'], str(alert_date), str(diff), d)
  File "bot.py", line 47, in create_hive10_dict
    send_sms(*args)
  File "bot.py", line 116, in send_sms
    tags = (alert_json['tags'][0])
IndexError: list index out of range

So I figured, adding "elif" statements between;

severity = (alert_json['severity']) # I want severity
          tags = (alert_json['tags'][0])
          tags_two = (alert_json['tags'][1])
          sourceRef = (alert_json['sourceRef'])
          source = (alert_json['source'])
          title = (alert_json['title'])
          description = (alert_json['description'])
          all_artifacts = (alert_json['artifacts']) # I want all artifacts
          data_two_artifacts = (alert_json['artifacts'][1]['data'])
          dataType_two_artifacts = (alert_json['artifacts'][1]['dataType'])
          data_artifacts = (alert_json['artifacts'][0]['data']) # I want only the data item which lies under the artifacts element
          dataType_artifacts = (alert_json['artifacts'][0]['dataType'])

between variables shown above might fix this, if something is not present in the alert, skip it etc. I have no idea how to it, still trying to find a way because the send_sms has else statement in it trying to find a way.

Thanks for your help Joshua, you were really helpful.

JoshuaSmeda commented 4 years ago

Hi @devatnull,

IndexError: list index out of range - this error means you are attempting to access a element of the list that doesn't exist.

You can take out the [0] of the tags = (alert_json['tags'][0]) and see if it makes a difference.

If you want to get a list of tags to be sent, you can use list comprehension to solve the problem, something like:

tags_list = []
for tag in alert_json['tags']:
  tags_list.append(tag)

Your tags list should look like this for example:

tags = ["Joe", "Zoe", "Brad", "Angelina", "Zuki", "Thandi", "Paris"]

Then just run this:

pretty_tags = ' '.join([x for x in tags_list])

print(pretty_tags)

Result:

Joe Zoe Brad Angelina Zuki Thandi Paris

This way you can collect all your tags without explicitly defining variables that may not exist (and therefore throw errors).

Otherwise, you're just going to have to do checks to see if there is a data in the variables before you send to Twilio. Something like:

if len(variable) <= 1:
  #has data
else:
  #has no data
  pass

devatnull commented 4 years ago

It works like a charm, if something is not present in the alert, it gets thrown out, only problem left is with the twilioAPI, sometimes the script makes it like the SMS is sent but in reality twilio debugger throws error code 30008, I have no idea why, I tried troubleshooting since your last comment, but it is fine, I really dont need the other stuff.

I hope they pay you well Joshua, you are an amazing thinkerer. Thanks for your help.

JoshuaSmeda commented 4 years ago

Awesome, that's good to hear :)

Yes the 30008 error could mean alot of things and might be challenging to debug. It's a issue on Twilios side.

Good luck friend!

On Thu, 30 Jul 2020, 18:04 devatnull, notifications@github.com wrote:

It works like a charm, if something is not present in the alert, it gets thrown out, only problem left is with the twilioAPI, sometimes the script makes it like the SMS is sent but in reality twilio debugger throws error code 30008, I have no idea why, I tried troubleshooting since your last comment, but it is fine, I really dont need the other stuff.

I hope they pay you well Joshua, you are an amazing thinkerer. Thanks for your help.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/JoshuaSmeda/TheHive_SLA_Monitor/issues/1#issuecomment-666490254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJRBWDTGW3MDD5CMAPDYL3LR6GKZLANCNFSM4PF3YJBA .