search

mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources
https://www.mlsecproject.org/
GNU General Public License v3.0
652 stars 179 forks source link

Winnow speedup #103

Closed krmaxwell closed 9 years ago

krmaxwell commented 9 years ago

Will post detailed comparisons from cProfile shortly. Ready for testing otherwise.

krmaxwell commented 9 years ago

For the record: on my R&D system, previously IP enrichment involving no DNSDB lookups (just local) didn't complete in at least 6 hours.

Just now:

real    2m42.864s
user    2m35.437s
sys 0m5.183s
krmaxwell commented 9 years ago

This should also fix #42.

krmaxwell commented 9 years ago

This seems to fix #104 and is ready for testing. For future reference, from the docs for the Python csv module:

Note: This version of the csv module doesn’t support Unicode input.

This means we have a new dependency. Sorry :disappointed: