search

mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources
https://www.mlsecproject.org/
GNU General Public License v3.0
652 stars 179 forks source link

cache geoip for better performance #93

Closed jeffbryner closed 9 years ago

jeffbryner commented 9 years ago

Cache the geoip .dat file to get better performance.

krmaxwell commented 9 years ago

My :trollface: succeeded

alexcpsec commented 9 years ago

Thanks, @jeffbryner ! I need to sort out the CLA and send it to you before merging. This has been a crazy week but I think I can manage it early next week.

alexcpsec commented 9 years ago

Just waiting on the CLA for this.

krmaxwell commented 9 years ago

Doing some testing. First, without this change (but including my other winnower speedups):

(venv)kmaxwell@newton:~/src/combine$ time python winnower.py
...
2015-01-07 18:05:29,667 - combine.winnower - INFO - Dumping results

real    2m30.208s
user    2m26.651s
sys 0m3.159s

Then, using pygeoip.MEMORY_CACHE:

(venv)kmaxwell@newton:~/src/combine$ time python winnower.py
...
2015-01-07 18:08:41,448 - combine.winnower - INFO - Dumping results

real    1m58.829s
user    1m57.389s
sys 0m1.096s

Not insignificant, that's like a 20% speedup! Hopefully @jeffbryner can get the CLA back and we can merge.

alexcpsec commented 9 years ago

Jeff just signed :shipit:

krmaxwell commented 9 years ago

:tada: