minemeld isn't applying 'tags' filter to CIF requests

[sfinlon]

In minemeld under the CIF miner I see options for tags, but the actual client doesn't appear to utilize the tags for filtering in the CIF request. https://github.com/PaloAltoNetworks/minemeld-core/blob/master/minemeld/ft/cif.py#L155

The tags should be sent with the request to filter, and reduce the time/size of the results.

[jtschichold]

Hi @sfinlon, This is possible today by specifying in the Miner config the filters to be sent to CIF: https://github.com/PaloAltoNetworks/minemeld-core/blob/master/minemeld/ft/cif.py#L40

[sfinlon]

I only question it because it doesn't appear to be pulling the full list that it should.

I have several miners with different configurations that I'm testing, and comparing them to the manual queries of the CIF client.

In one example, my filters are: "FILTERS | confidence: 75 otype: ipv4 tags: malware, scanner, botnet, exploit, phishing" I get the following results: "ADDED: 886 REMOVED: 0 | RX: 0 PROCESSED: 0 TX: 3446"

So it only shows 886 from the server, but if I run the following directly on the CIF client: "cif --feed --otype ipv4 -c 75 --tags=malware,scanner,botnet,exploit,phishing" It returns 30,082 IPs.

I'm not sure how or why there would be a 29,000 indicator difference.

[sfinlon]

Ok, so I created a brand new Miner, as the other one looked to have been set to one setting and then changed. In the new Miner I set "confidence: 75 otype: ipv4 tags: malware, scanner, botnet, exploit, phishing" and refreshed and it pulled 30,097 IPs.

So I guess my question is if you need to modify the filters in a Miner, do you have to create a new Miner in order to get the the results you need? The difference in IPs 30,097 to 30,082 between Minemeld and the CIF client I am going to attribute to lack of whitelisting which I reference in Issue #7

[jtschichold]

Hi @sfinlon, the Miner polls indicators added from the last poll. Only the first time you create the Miner, the Miner will go back 7 days (configurable via the 'initial_days' parameter).

[sfinlon]

So if I create a miner and set it to confidence 75 and tags of botnet,exploit, and then decide to up the confidence to 85 and add the tag malware, it doesn't modify the list it's pulled at all. I have to delete that initial miner and create a brand new one to pull the correct list?

[jtschichold]

Hi @sfinlon, yes, that's correct. There is a workaround for this but you have to do it via the shell: mm-console signal flush

luigi