search

HurricaneLabs / machinae

Machinae Security Intelligence Collector
MIT License
502 stars 99 forks source link

GeoIP within URL search #27

Open Den1al opened 8 years ago

Den1al commented 8 years ago

When triggering a search on a URL (with -O url), the script dosen't search for geo ip locations of the associated ips of the domain.

mcm commented 8 years ago

Hmm yes that's right. It doesn't for a domain either. How would you expect it to work? Should it run all of the sites for the related domain and IPs?

Steve McMaster Security Operations Lead mcmaster@hurricanelabs.com Direct: 216.923.1330 x108 <216.923.1330;108> SOC: 216.923.1330 x4

On May 4, 2016 at 8:40:51 AM, Den1al (notifications@github.com) wrote:

When triggering a search on a URL (with -O url), the script dosen't search for geo ip locations of the associated ips of the domain.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/HurricaneLabs/machinae/issues/27

Den1al commented 8 years ago

The output gets you various associated ip addresses. It would be ideal for the script for at least give a geo location of one of them.

mcm commented 8 years ago

@Den1al So what if there were a command line option -R that were the "recursive" sites you wanted to run. For example, if you passed a URL and said -R freegeoip you would get the sites for the URL, but also the freegeoip for all IP addresses the domain resolves to. Or you could pass -R malc0de with a URL and it would run malc0de for the domain and all IP's.

I don't want to hardcode anything like "run GeoIP for the IP for a URL" because other users may not have a GeoIP site, or may have a different one.

Den1al commented 8 years ago

Sound like an excellent idea.

mcm commented 8 years ago

@Den1al Can you give this a test and let me know? I've added it in the mcm/gh_issue_27 branch, I think you can install that like this but I might be totally wrong:

pip3 install -U git+https://github.com/HurricaneLabs/machinae.git@mcm/gh_issue_27
mcm commented 8 years ago

I suppose usage info would've been useful...

~/g/h/machinae ❯❯❯ machinae -s none -R freegeoip google.com
[.] Requesting https://freegeoip.io/json/google.com (GET)
********************************************************************************
* Information for google.com
* Observable type: fqdn (Auto-detected: True)
********************************************************************************

[+] freegeoip.io results
    [-] GeoIP Country Code: US
    [-] GeoIP Country: United States
    [-] GeoIP Region Code: CA
    [-] GeoIP Region: California
    [-] GeoIP City: Mountain View
    [-] GeoIP Zip Code: 94043
    [-] GeoIP Latitude: 37.4192
    [-] GeoIP Longitude: -122.0574
    [-] GeoIP Timezone: America/Los_Angeles
~/g/h/machinae ❯❯❯
greggel commented 5 years ago

Do you have usage info on how all of the flags could be used? This example does not work for me either.

mcm commented 5 years ago

@greggel - couple things. First, it looks like we blew away that branch without ever actually merging it. It exists again, and I did have to fix something to make it work, so perhaps that's what you were running into. But also, at least from here, it looks like freegeoip isn't working at the moment.

I tested again with the ipwhois site instead, which seems to be working:

(machinae_issue_27) [machinae_issue_27] /t/machinae_issue_27 ❯❯❯ machinae -s none -R ipwhois hurricanelabs.com
********************************************************************************
* Information for hurricanelabs.com
* Observable type: fqdn (Auto-detected: True)
********************************************************************************

[+] IP Whois results
    [-] ASN Information: ('30148', '192.124.249.0/24', '2015-04-01', 'arin', 'US')
    [-] Network Information: ('192.124.249.0/24', 'NET-192-124-249-0-1', 'SUCURI-ARIN-002', '192.124.249.0 - 192.124.249.255')
    [-] Registration Info: ('Sucuri', '2015-04-01', '2015-04-01')
    [-] Registration Locality: ('Menifee', 'CA', '92584', 'US')
    [-] Abuse Email: soc@sucuri.net
    [-] Tech Email: soc@sucuri.net
(machinae_issue_27) [machinae_issue_27] /t/machinae_issue_27 ❯❯❯

Try installing from that branch in git again if you can (sorry you're going to also lose like 2 years of features but IF that works we can get the branch incorporated and released) and give it another test. If that doesn't work, can you share the output you get when it's not working?

greggel commented 5 years ago

Maybe its because I am on python 3.7? I am getting a 'unable to find magic library' currently. What version of python is used here?

mcm commented 5 years ago

Hmm.... I'm running it on Python 3.7 locally as well. What OS platform are you on? We're using the Python library filemagic which has a requirement for libmagic to exist at the system level: https://pypi.org/project/filemagic/ -- can you confirm whether that's installed? If not, we can add a note to the README about that requirement, and possibly catch the error you're getting to display a more friendly error.

greggel commented 5 years ago

That https://pypi.org/project/filemagic/ doc was helpful! I had installed filemagic with pip but not libmagic with brew. Installing libmagic with brew fixed the issue and its all now working. I am running MacOS Sierra. And you were right about freegeoip giving a service temporarily unavailable but ipwhois is currently working. Thanks!

mcm commented 5 years ago

Awesome! We'll get these changes merged and a new release published.