search

rocknsm / rock-dashboards

Dashboards and loader for ROCK NSM dashboards
http://rocknsm.io
Apache License 2.0
47 stars 17 forks source link

Update logstash-850-filter-ip_addresses-enrich.conf #59

Closed peasead closed 4 years ago

peasead commented 4 years ago

For #58

neu5ron commented 4 years ago

so based on the log/background for #58 - there appears to be a wild/rare value of ["urn:uuid:A92F6F89-B656-4DAD-B0F1-FE50FC973A22"] coming from one of the field names that gets renamed to client.ip. I can't see any fields that would cause this in the current logstash configs, basically there isn't a field that rarely is an IP thats getting renamed to client.ip.

anyways. removing this portion, at the end of the pipeline you would now have the value ["urn:uuid:A92F6F89-B656-4DAD-B0F1-FE50FC973A22"] trying to be ingested into the database which will error out and if it doesn't error out thats bad and elasticsearch has a really bad IPv4 or IPv6 verification - which I don't even want to start down that road at the moment........

so couple questions:

  1. how often does this occur?
  2. do you know what log source caused this?
  3. any pcap specific that you are seeing this happen in? or was this just a log that came across after running traffic for some time?
peasead commented 4 years ago

so based on the log/background for #58 - there appears to be a wild/rare value of ["urn:uuid:A92F6F89-B656-4DAD-B0F1-FE50FC973A22"] coming from one of the field names that gets renamed to client.ip. I can't see any fields that would cause this in the current logstash configs, basically there isn't a field that rarely is an IP thats getting renamed to client.ip.

anyways. removing this portion, at the end of the pipeline you would now have the value ["urn:uuid:A92F6F89-B656-4DAD-B0F1-FE50FC973A22"] trying to be ingested into the database which will error out and if it doesn't error out thats bad and elasticsearch has a really bad IPv4 or IPv6 verification - which I don't even want to start down that road at the moment........

so couple questions:

  1. how often does this occur?
  2. do you know what log source caused this?
  3. any pcap specific that you are seeing this happen in? or was this just a log that came across after running traffic for some time?

Yep, you're right on regarding the uuid. It's different with each entry, but I wanted to put the whole event there.

  1. how often does this occur?
    • about 4-6x's per second
  2. do you know what log source caused this?
    • I don't, how could I figure that out? I'm also not convinced logstash-850-filter-ip_addresses-enrich.conf is the only file that's causing it.
  3. any pcap specific that you are seeing this happen in? or was this just a log that came across after running traffic for some time?
    • it's running right now, but I can't provide the pcap
peasead commented 4 years ago

I'm going to close this...I just tried this off the pre-release candidate and I'm not seeing it.