search

certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
https://docs.intelmq.org/latest/
GNU Affero General Public License v3.0
977 stars 296 forks source link

BOT: DNS lookup #373

Closed robcza closed 4 years ago

robcza commented 9 years ago

We are considering a bot to do a DNS lookup for events, where only fqdn is included. Except the performance issues that will be the same as in the case of reverse_dns expert bot, we have encountered another issue. Forward lookup could return more than one IP address for the domain, what to do with the result?

What do you suggest?

aaronkaplan commented 9 years ago

On 22 Oct 2015, at 15:03, Robert Šefr notifications@github.com wrote:

We are considering a bot to do a DNS lookup for events, where only fqdn is included. Except the performance issues that will be the same as in the case of reverse_dns expert bot, we have encountered another issue. Forward lookup could return more than one IP address for the domain, what to do with the result?

hmm, good question.

• split the event in more events and fill the source.ip with the results of the dns lookup

I guess that depends on the event.

• add new attribute for the results of the dns lookup an array?

What do you suggest?

Can you give us an example of such an event?

— Reply to this email directly or view it on GitHub.

robcza commented 9 years ago

Array is an obvious solution, and although I'm not a fan of adding new attributes, I don't think we should split an event into more. The logic behind the event is "this fqdn is bad" and splitting the event would be misleading on the output. My opinion is - we should add new array attribute.

I do not have any particular event example, but any domain with more IPs will do, sorry google:

Non-authoritative answer:
Name:   google.com
Address: 173.194.123.72
Name:   google.com
Address: 173.194.123.68
Name:   google.com
Address: 173.194.123.73
Name:   google.com
Address: 173.194.123.67
Name:   google.com
Address: 173.194.123.69
Name:   google.com
Address: 173.194.123.65
Name:   google.com
Address: 173.194.123.70
Name:   google.com
Address: 173.194.123.66
Name:   google.com
Address: 173.194.123.71
Name:   google.com
Address: 173.194.123.78
Name:   google.com
Address: 173.194.123.64
e3rd commented 9 years ago

What about 'forward_dns' to be the name? Note that 'reverse_dns' exists just for the opposite case.

aaronkaplan commented 9 years ago

On 22 Oct 2015, at 15:39, Robert Šefr notifications@github.com wrote:

Array is an obvious solution, and although I'm not a fan of adding new attributes, I don't think we should split an event into more. The logic behind the event is "this fqdn is bad" and splitting the event would be misleading on the output. My opinion is - we should add new array attribute.

I do not have any particular event example, but any domain with more IPs will do, sorry google:

Got it. Okay, I came across the same problem once when thinking about pDNS. So, if we have an array, then potentially we have a very large array. Still I prefer an array to splitting the event.

Non-authoritative answer: Name: google.com Address: 173.194.123.72 Name: google.com Address: 173.194.123.68 Name: google.com Address: 173.194.123.73 Name: google.com Address: 173.194.123.67 Name: google.com Address: 173.194.123.69 Name: google.com Address: 173.194.123.65 Name: google.com Address: 173.194.123.70 Name: google.com Address: 173.194.123.66 Name: google.com Address: 173.194.123.71 Name: google.com Address: 173.194.123.78 Name: google.com Address: 173.194.123.64

— Reply to this email directly or view it on GitHub.

aaronkaplan commented 9 years ago

On 22 Oct 2015, at 15:42, e3rd notifications@github.com wrote:

What about 'forward_dns' to be the name? Note that 'reverse_dns' exists just for the opposite case.

Makes sense for me as well.

e3rd commented 8 years ago

I've been trying to implement this, however, when configuring forward_dns in harmonization.conf, I am unable to set the type as "list". I'd like to put it as "list of IPAddress" but this seems to not be possible. I have to set forward_dns as String and join the list of IPs to the signle String.

There is no "list" type option in harmonization, or am I blind?

sebix commented 8 years ago

@e3rd Yes this is correct. We had a discussion on a similar problem concerning email addresses (abuse contact), see #185. But this is a different case, as data is derived from IP addresses, but not from abuse contacts.

Alternative to lists: Clone the event and create duplicates for each address. I prefer this variant as I see severe issues when saving multiple IPs per event. Small example: Given is a domain used as C&C server or a websites which distributes malware. The FQDN resolves to multiple addresses in different AS' and countries. We do ASN and CC lookups, but how to save these informations? How to filter them? Having non-distinct data in such fundamental fields means events are not unique anymore. With multiple addresses per event, how to save data which depends on it? Lists of CCs, ASNs etc? Needs big adoptions everywhere.

For all others: The forward DNS bot proposal can be found here:

robcza commented 8 years ago

Ok, cloning the events makes sense. I agree with @sebix @e3rd could you please adjust the bot accordingly?

e3rd commented 8 years ago

Ok, I'll remake it and pull it here.

e3rd commented 8 years ago

Interesting problem that I do not dare solve without your consent. There may be M ip addresses related to source and N addresses related to the destination. Should I make: A) create lot of events: M × event (source.ip = ..., dest.ip = null) + N × event (source.ip = null, dest.ip = ...) B) randomly map source to destination: M × event (source.ip = ..., dest.ip = ...) + (M-N) × event (source.ip = ..., dest.ip = null) C) ...?

Did you get the point of the problem, am I clear?

aaronkaplan commented 8 years ago

On 25 Nov 2015, at 12:28, e3rd notifications@github.com wrote:

Interesting problem that I do not dare solve without your consent. There may be M ip addresses related to source and N addresses related to the destination. Should I make: A) create lot of events: M × event (source.ip = ..., dest.ip = null) + N × event (source.ip = null, dest.ip = ...) B) randomly map source to destination: M × event (source.ip = ..., dest.ip = ...) + (M-N) × event (source.ip = ..., dest.ip = null) C) ...?

Did you get the point of the problem, am I clear?

I understand and I am not sure if it’s solvable. I believe we are missing information here (e.g. net flow data which would prove which src ip talked to which dst ip). Inventing an event is probably not a good solution.

It’s a bigger architecture discussion IMHO. Let’s have a call on this.

Best, a.

aaronkaplan commented 8 years ago

will be looked at in Release-2 or -3

e3rd commented 4 years ago

I think this is partly covered by Gethostbyname bot and we may close this issue. In most of the cases, single IP address produced from a FQDN is enough.