search

PaloAltoNetworks / minemeld

Main MineMeld documentation repo
380 stars 64 forks source link

Performance problem when stdlib.listIPv4Generic node contains a large number of addresses #31

Closed pgerencser closed 5 years ago

pgerencser commented 5 years ago

We would like to use a stdlib.listIPv4Generic miner with more than 25000 addresses. These addresses has been imported from a texfile using minemeld-sync.py. My problem is that it takes around 20 seconds to list the indicators (500 pages, 50 indicators per page). During the process minemeld generates the paginator component, one CPU core is used at 100%, but other cores are not used. When I add a new IP, after ~10 seconds I get a green message saying "Changes saved". When I refresh the indicator list, I can find the recently added address. During the add process one core of the CPU works at 100% for several sconds. So at only one address it works, just a bit slow. When I add 3-4 IPs one-by-one using the web gui (my typing speed is around 3 seconds per IP), I get a red message: "Error adding indicator: Timeout". In this case recently added addresses has not got in the list.

jtschichold commented 5 years ago

@pgerencser which type of Miner are you using to add indicators? Please make sure you are using stdlib.localDB Miner

pgerencser commented 5 years ago

@jtschichold Thank you for your reply, using stdlib.localDB miner makes it much more faster.