search

johnnykv / mnemosyne

Normalizer for honeypot data.
GNU General Public License v3.0
45 stars 40 forks source link

Batch incomming hpfeed data #10

Closed johnnykv closed 11 years ago

johnnykv commented 11 years ago

HPFeed data needs to be batched up before inserting into mongodb. This will resolve locking issues when doing full renormalizations. The queue need to be implemented around here: https://github.com/johnnykv/mnemosyne/blob/master/persistance/mnemodb.py#L98

At the current load it seems appropriate to process the queue every 30 second, which gives around 100 entries to process.

johnnykv commented 11 years ago

Not relevant anymore. Optimisation of indexes seems to have resolved the issue.