search

chrismaddalena / Goreport

A Python script to collect campaign data from Gophish and generate a report
BSD 3-Clause "New" or "Revised" License
180 stars 50 forks source link

No Error Reasoning #11

Closed caseyagil closed 5 years ago

caseyagil commented 7 years ago

I am running a campaign report that has 157 "users" however I get a,

" [!] There was a problem processing campaign ID *** " L.. Details: 'location'

I am not sure why this is happening except that maybe I am hitting the thresh hold request through the IP location API's. Is there a way to tell it to slow down? Like I see it says 50 requests a second and that there seems to be an issue with getting ip locations... Thoughts / Help? Attaching a photo as well. screen shot 2017-11-06 at 11 36 25 am

chrismaddalena commented 7 years ago

Hey there, this error's details aren't super helpful, but I can tell you what it means. There's no 'location' key in the results here. There's one place a 'location' key is referenced and it is when looking up the geolocation of an IP address with MaxMind. We can run into problems if GoReport asks for a geolocation back-to-back too quickly, but GoReport should be staggering requests enough to avoid that and usually that problem comes from Google's API. Even when we do hit a limit, something should be returned. It seems in this case MaxMind returned a result without the 'location' key, so GoReport's attempt to get the coordinates failed.

I haven't run into this before, even with tons of requests for geolocation lookups. It's possible something changed that I will now need to account for. I've been planning to fix-up the geolocation bits and should tackle that soon. For now I think I can add a simple check to make sure the key exists before getting the contents. That should address this problem and allow the campaign processing to continue instead of failing because one lookup for one event failed. GoReport can't just assume this key exists anymore, apparently.

I'll get that change in shortly, like later today.

caseyagil commented 7 years ago

Ok, sounds great. After tinkering a few hours, for some reason I just deleted the user for ID 66. And the whole thing ran fine. Still unclear as to what might had happened since some of my users had no location while others did, yet that particular user was the issue?

Would it be possible to get an explanation or capability to disable the function for the IP / Geolocating? I tried to carve out that portion of code on my own but I am still learning and had many issues.

chrismaddalena commented 7 years ago

Sorry for the big delay. I just committed some basic error handling for the Geolite2 lookups. It does seem strange that just one IP address was such an issue when the rest were fine. If the issue could be reproduced, some additional data could be collected with more verbose error details. Right now, the errors are generic, so sometimes the details are... more-or-less non-existent.

What I can tell you now is this line is the issue:

mmdb_latitude, mmdb_longitude = mmdb_location['location']['latitude']

These lines are the only lines that reference a key named 'location'. Any future errors shouldn't cause the report to fail.

I like the idea of adding an option to disable these checks. I'll work on that. One reason GoReport run these checks is to double-check GoPhish's geolocation results. I've found them to be way off in some instances, but a location would still be available for the report even if the checks are disabled. It'll be a good enhancement.

chrismaddalena commented 5 years ago

Closing this because the offending functions have been entirely removed and replaced with geolocation options that are more reliable, easier to troubleshoot, and more efficient/faster. This and much, much more is included in v3.0 just pushed to master.