mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources
https://www.mlsecproject.org/
GNU General Public License v3.0
653 stars 179 forks source link

Enrich DNS indicators #36

Closed krmaxwell closed 9 years ago

krmaxwell commented 10 years ago

LeaveNoIndicatorBehind

krmaxwell commented 9 years ago

So, um. What do we want to do on this one?

alexcpsec commented 9 years ago

So here is the original spec of what I wanted to do for the tiq-test enrichment format.

We would get something like this from the raw data:

"entity","type","direction","source","notes","date"
"bgr.runk.pl","FQDN","outbound","mtc_malwaredns","Malware","2014-06-01"

And turn it into (getting some random IPs and enrichments to illustrate):

"entity","type","direction","source","notes","date","asnumber","asname","country","host","rhost"
"31.210.109.42","IPv4","outbound","mtc_malwaredns","Malware","2014-06-01","9318","Hanaro Telecom Inc.","XX","bgr.runk.pl","whatever.rdomain.com"
"5.63.155.203","IPv4","outbound","mtc_malwaredns","Malware","2014-06-01","2343","ASN for what?","XX","bgr.runk.pl","whatever.rdomain2.com"

So in this "enrichment type", we are transforming the FQDN entries into enriched IPv4 entries doing the following:

1) We run the FQDN entity through the Passive DNS database (an A query), limiting the IP response to the date of the indicator. So algorithmically we only return IPs where (firstseen <= $date 23:59:59) && (lastseen >= $date 00:00:00)

2) These IPs are used to create new IPv4 entries where source, notes and date are the same as the original FQDN entry, and the host column is populated by the original FQDN entity.

3) These new IPv4 entries are enriched using the process we already defined for enrich_ip to add asnumber, asname, country and rhost

Of course this is all moot if there is no DNSDB key. In that case, all FQDN entries should be dropped on the enrichment.

krmaxwell commented 9 years ago

What we currently lack is (3), I believe. We already do (1) and most of (2), though it needs a bit of extension to be complete.

alexcpsec commented 9 years ago

Let me know if I can help.

krmaxwell commented 9 years ago

In the final, enriched data, does the FQDN still have its own entry, or are we just using the IPv4 entry? In other words, does the original FQDN IOC map to :one: or :two: enriched IOCs? I think it should be two, myself.

krmaxwell commented 9 years ago

nvm I figured out where everything goes. :blush: