certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
https://docs.intelmq.org/latest/
GNU Affero General Public License v3.0
975 stars 296 forks source link

Cymru parser_cap_program invalid hostname filtering #2169

Closed ClevenL closed 2 years ago

ClevenL commented 2 years ago

In some Team-Cymru payloads the optional_information entry includes hostname: parameter with an IP address. This is parsed in intelmq.bots.parsers.cymru.parser_cap_program bot into source.fqdn field, which does not allow IP entries.

There should be check to filter out IP hostnames, since we already have that data included in source.ip field.

The fix can be equivalent to PR #2144. Including FQDN.is_valid check seems overkill in this case, but I included it in the example below.

elif key == 'hostname':
    # some hostnames are IP and not valid fqdn
    if not FQDN.is_valid(value) and value == ip:
        continue
    event['source.fqdn'] = value
gethvi commented 2 years ago

Perhaps cleaner way:

event.add("source.fqdn", value, raise_failure=False)
sebix commented 2 years ago

Perhaps cleaner way:

event.add("source.fqdn", value, raise_failure=False)

But that could possibly hide other errors as well (e.g. other kind of data, wrong parsing, etc.)

So I prefer @ClevenL's solution.

gethvi commented 2 years ago

But that could possibly hide other errors as well (e.g. other kind of data, wrong parsing, etc.)

Would it make sense to modify the add function with logging the error instead of just returning False?

https://github.com/certtools/intelmq/blob/6e1c1bf07b733e5ea3365a74e23105cc613450f9/intelmq/lib/message.py#L259-L264