Issue with ush_zmq_to_influxdb.py

[manosmustang]

Hi,

it seems that in the influxdb I do not get all the measurements. current MISP version that I have is v2.4.172

Do I need to change anything on the push_zmq_to_influxdb.py file?

[manosmustang]

I added the below parts to push_zmq_to_influxdb.py and now I can see everything

    if topic == "misp_json" and "Event" in m:
        logging.info("misp_json Event pushed to InfluxDB")
        event = m["Event"]
        api.write(bucket="misp", record={
            "measurement": "event_details",
            "tags": {
                "instance": instance,
                "event_id": event.get("id", ""),
                "org_id": event.get("org_id", ""),
                "orgc_id": event.get("orgc_id", "")
            },
            "fields": {
                "info": event.get("info", ""),
                "published": str(event.get("published", False)),
                "attribute_count": event.get("attribute_count", 0),
                "analysis": event.get("analysis", ""),
                "distribution": event.get("distribution", ""),
                "threat_level_id": event.get("threat_level_id", "")
            },
            "time": int(float(event.get("timestamp", recv_ts)) * 1000000000)
        })

    if topic == "misp_json_user":
        logging.info("misp_json_user pushed to InfluxDB")
        user = m.get("User", {})
        org = m.get("Organisation", {})
        action = m.get("action", "")
        api.write(bucket="misp", record={
            "measurement": "user_activity",
            "tags": {
                "instance": instance,
                "user_id": user.get("id", ""),
                "org_id": user.get("org_id", ""),
                "action": action
            },
            "fields": {
                "email": user.get("email", ""),
                "last_login": user.get("last_login", ""),
                "org_name": org.get("name", ""),
                "org_type": org.get("type", ""),
                "org_uuid": org.get("uuid", "")
            }
        })

[manosmustang]

I have made also from my side a change at telegraf.conf file

The grok pattern in my case was not working so this is the one that is working:

[[inputs.tail]]
  files = ["/var/log/apache2/misp.local_access.log"]
  name_override = "misp_apache_access_log"
  data_format = "grok"
  grok_patterns = [
    "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?\" %{NUMBER:response:int} (?:%{NUMBER:bytes:int}|-) %{QS:referrer} %{QS:useragent}",
    "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"-\" %{NUMBER:response:int} (?:%{NUMBER:bytes:int}|-) \"-\" \"-\""
  ]

[kanekennedy]

@manosmustang

Hi, I tried your grok pattern but my telegraf fails to start with it, has the copy and paste garbled this part?

(?:%{NUMBER:bytes:int}|-) %{QS:refe>    "%{IPORHOST:clientip}

Have you made any changes to the error log grok patteren as well, my dashboard still only shows me ZMQ uptime and I am not even sure that is correct :(

[manosmustang]

@manosmustang

Hi, I tried your grok pattern but my telegraf fails to start with it, has the copy and paste garbled this part?

(?:%{NUMBER:bytes:int}|-) %{QS:refe>    "%{IPORHOST:clientip}

Have you made any changes to the error log grok patteren as well, my dashboard still only shows me ZMQ uptime and I am not even sure that is correct :(

Hi @kanekennedy , yeah, issue with paste. I have updated, please check it now

[manosmustang]

@kanekennedy

Remove also the "#" from "debug = true" in order to check if you actually receive data

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = ""
  omit_hostname = false
  debug = true
  # quiet = false