cif-smrt performance

[Nibor62]

Hi,

I am currently testing CIF (master branch) and I have issues with cif-smrt. Its updates are quite slow and with source like Alienvault I get an error after ~15min while CPU usage stay really low (<10%). After Alienvault fails, all the other feeds goes into the same error (509). I investigated a bit and it seem that starman is doing a lots of requests in background. So,

• Is it a known issue ?
• I have seen that "enable_metadata" may disable somethings but I haven't found any documentation about it. And I still have a lot of requests (mostly AS number) getting sent. What is "enable_metadata" doing exactly ?
• Is there any way to disable completely post working on data ?

[wesyoung]

• it's a known feature, resolving things like ASN and Geo info for each IP. if you tcpdump the network traffic you should see lots of activity
• enable_metadata isn't really that doc'd because not a lot of people use it, disabling it should get you the desired result (a default we're looking at changing on v3). it's very useful information, but we've seen with smaller deployments not everyone wants that type of data in their instance (or cares).

[Nibor62]

Thank for the answer,

Actually I don't need this kind of information but putting enable_metadata in the conf doesn't seem to stop ASN request (still see them passing through with tcpdump). How am I supposed to disable it ?

[wesyoung]

iirc (it's been a while); try modifying /etc/cif/cif-starman.conf to look like:

{
#       "tlp_map" => {
#               "red" => "restricted",
#               "amber" => "privileged",
#               "green" => "limited",
#               "white" => "public"
#       }
         "enable_metadata" => '0',
}

and restart cif-starman.

this should be picked up here:

https://github.com/csirtgadgets/massive-octo-spice/blob/a6994e8725a06daaeea8f70578f0322f96346e72/src/lib/CIF/REST.pm#L68

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Client.pm#L258

and keep the plugins from being called here:

https://github.com/csirtgadgets/massive-octo-spice/tree/develop/src/lib/CIF/Meta

ref:

https://github.com/csirtgadgets/massive-octo-spice/blob/f9e85d2c1af7e8d3cf77b0f0b86595dce0bcf0a6/hacking/platforms/ubuntu/cif-starman.conf

[Nibor62]

aarrg, this language is giving me headache ! Well, actually, at this line : https://github.com/csirtgadgets/massive-octo-spice/blob/a6994e8725a06daaeea8f70578f0322f96346e72/src/lib/CIF/REST.pm#L68 You are using the name 'enable_meta_data' which is named 'enable_metadata' here: https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/lib/CIF/Client.pm#L35

If i correctly understand the darks incantations behind moose, it is a bug isn't it ?

I have tested to rename it and it effectively prevent "_process_metadata" to get called and bring a nice performance improvement (and actually allow me to parse alienvault without error). I still have a lot of DNS request passing through, which came from cif-worker (after cif-smrt update finished). Any configuration options for this one ?

[wesyoung]

don't remember. might be a typo that "just stuck". like i said; most instances leverage the data so those sections haven't been all that refined (at-least in v2, v3 is a bit diff by default).

the dns req's are coming from cif-worker, which ... if you don't want that data, don't start that service.

[Nibor62]

Well it's working thank you. However there is still an issue with enable_meta_data naming. PL incoming