Adding data into CRITs

[shiv183]

I followed everything as mentioned for combine installation. I did configuration with CRITs API and port number also.

Combine could able to get both inbound and outbound data into harvester.json

But when I run ./combine.py -t crits, i see ("POST /api/v1/ips HTTP/1.1" 200 XXX ). When fetching part is done, and check into CRITS, I found no data from combine campaign.

Tell me what's the exact issue I'm facing. I just want data to be fetched into CRITs.

Thanks in advance!

[alexcpsec]

Hi, @shiv183 !

Are there log entries from CRITs that you can review? Also, have you created the "combine" campaign on CRITs beforehand?

[shiv183]

Hi, @alexcpsec

I've added a campaign "Combine"vinto CRITs before running combine.py.

Combine terminal confirms that "combine.baler - INFO - Output 2461 indicators to CRITs using 10 threads. Operation took 198 seconds"

While in CRITs terminal, I'm seeing ("POST /api/vi/ips/ HTTP/1.1" 200 205 ) for each indicators when it is added.

But I'm not able to find any data under campaign "Combine".

Help me out in this and Thanks in advance!

[mgoffin]

@shiv183 One thing I would check is to see if you have Source Access to the content being imported from Combine into your CRITs instance.

I can't say for sure what the sources are as you import them from Combine, but if your CRITs user hasn't been granted Source Access to them, then you'll never see that data even though it exists.

For example, if your CRITs user has access to source "Foo", but Combine sends an Indicator into CRITs from source "Bar", you won't be able to see it until you add source "Bar" to CRITs and then edit your user to have access to source "Bar".

[shiv183]

@mgoffin i added "combine" as both campaign and source but still couldn't able to see the data. i use admin account which is the only account in CRITs.

[mgoffin]

If you can get to a mongo shell on your machine, I'd like to know the output of the following:

use crits db.indicators.aggregate( [ { $group : { _id : "$source.name" } } ] )

If there are any sensitive source names you don't wish to share with us, feel free to leave them out. But my goal here is to see what source names are returned, and to make sure you've added each one to your CRITs install and given your admin account permission to access each one (being an admin doesn't give you access to them automatically, you have to manually give yourself access to them).

[shiv183]

@mgoffin db.indicators.aggregate( [ { $group : { _id : "$source.name" } } ] ) { "_id" : [ "www.malwaregroup.com" ] } { "_id" : [ "www.nothink.org" ] } { "_id" : [ "www.blocklist.de" ] }

The mongoDB doesn't show combine but i have added it to the source list you can see in the below screenshot. i thought that combine is case sensitive so i added two campaign as "Combine" and "combine" as well as source.

[mgoffin]

You won't be able to see any of those indicators until you add those strings as sources to your system and give yourself access to them.

That being said, it seems a bit broken to me that Combine would be using what look like indicator values as source names. It would make managing source access quite the nightmare. Is something in Combine misconfigured?

[krmaxwell]

It's not so much "misconfigured" as it is a design decision that I hope we change Soon :tm:. Those aren't indicator values by the way: they're the original sources of the data that Combine pulled.

[mgoffin]

Ahh. So then it's "working as expected" in this case. OP just needs to be able to figure out the sources that Combine is pulling data in with. Can he do that without having to do this query on a regular basis?

[shiv183]

@mgoffin , @krmaxwell ... Still to be more clear with you guys... In the dashboard, i can see number of ips added from campaign 'combine' is around 3k. When i check into combine campaign, no ips are listed.

Not sure is it some problem with CRITs or combine.

[paulpc]

@mgoffin, i wanted to add the sources automatically via Combine, but t the time, the API didn't allow for that (and I didn't know how to look for the sources otherwise and prompt the user that the sources need to be added with the appropriate perms). @shiv183 , i've bumped into this issue before when adding indicators with a campaign / source that hasn't yet been created in the CRITs GUI (I tried to fix the campaign issue in the last PR)

[shiv183]

@paulpc , @mgoffin , @krmaxwell .. Today i reworked everything with CRITs and combine. Added combine as CRITs, and indicators to sources. And i could see all the feeds pulled into CRITs. Thanks for your assistance guys.

But now i face different problem. All data that are generated from both inbound , outbound feeds has same source "www.blocklist.de" in CRITs.

[paulpc]

@shiv183, good job on getting it going. Sorry for crappy docs on my part on the CRITs integration.

As for the inbound / outbound, but not sure it's worth the time before @alexcpsec switches the main branch to @sooshie's code. Once you have more data from the feeds (inbound|outbound|bot c2|spam server| et cetera), it might be worth to add tags to the indicators, or better define the sources. For example, the metadata in the alienvault feed is awesome in regards to describing what the indicator is / does; once it's part of the object passed to the baler.py script, it would be easy(ish) to port it to CRITs.