Feature Request: Update Feed - generate_feed_from_raw_iocs.py

[bentedesco]

cbfeeds/example/raw/generate_feed_from_raw_iocs.py

Instead of only being able to create a new feed with a single report, it would be nice if we also had the functionality to update an existing feed and concatenate additional reports to the bottom.

[bentedesco]

@askthedragon

[0duck0]

up-vote!

Further details: If you look at commercial feeds, from the CbResponse UI, you will notice that they include multiple reports, each with it's own unique IOCs.

If I create a raw feed from my own flat file of IP addresses, for example: I would run the following command from the terminal to convert my flat file in to a json formatted feed file for CbR.

python /etc/cb/feeds/cbfeeds-master/example/raw/generate_feed_from_raw_iocs.py -n CbFeed -d "Carbon Black Custom Feed" -u "http://www.carbonblack.com" -s "This is a feed used to demostrate Cb custom feeds." -t "Not much to say on tech data." -i image/Cb.png -I ip_addresses.txt -r "Cb Report"

Now I would login to the UI, setup a feed, and point it to the path of the new json formatted feed file.

But... What if we found out about some new campaign or "actor" and I want to associate the new IPs with this new campaign to that feed. I would like to add this additional report to the feed that we already have. The only way I know how to add additional IOCs or in this case IP addresses to the feed, is by adding them to the original list or flat file that we used above and running the same command again. But this will only replace the original report with a new one. All that we need is another Python script that appends another report to the existing feed.

When I look at other feeds in their json format, I notice that they have multiple reports in them. Here is some of what is included in the ThreatConnect.json feed:

{ "feedinfo": { "provider_url": "http://www.threatconnect.com/", "display_name": "ThreatConnect Carbon Black Community", "name": "ThreatConnect", "feed_url": "https://api.alliance.carbonblack.com/feed/ThreatConnect", "summary": "Threat intelligence data provided by ThreatConnect to the Carbon Black Community", "tech_data": "There are no requirements to share any data to receive this feed.", "order": 99, "icon": "" }, "reports": [ { "title": "Malicious binary associated with likely Pakistani cyber espionage.", "timestamp": 1401326988, "iocs": { "md5": [ "165AC370B54E664812E4C15B2396CCD6" ] }, "score": 100, "link": "https://app.threatconnect.com/tc/auth/indicators/details/file.xhtml?file=165AC370B54E664812E4C15B2396CCD6&owner=Common+Community", "id": "79876" }, { "title": "Malicious binary associated with likely Pakistani cyber espionage.", "timestamp": 1401326988, "iocs": { "md5": [ "35663E66D02E889D35AA5608C61795EB" ] }, "score": 100, "link": "https://app.threatconnect.com/tc/auth/indicators/details/file.xhtml?file=35663E66D02E889D35AA5608C61795EB&owner=Common+Community", "id": "79877" }, { "title": "Malicious binary associated with likely Pakistani cyber espionage.", "timestamp": 1401326988, "iocs": { "md5": [ "A21F2CB65A3467925C1615794CCE7581" ] }, "score": 100, "link": "https://app.threatconnect.com/tc/auth/indicators/details/file.xhtml?file=A21F2CB65A3467925C1615794CCE7581&owner=Common+Community", "id": "79878" }, ... TRUNKATED

[askthedragon]

This is good feedback. We typically put each IOC in their own threat report because that is usually what is asked, but maybe allowing an option to consolidate all IOCs into one threat report is needed.

[0duck0]

Actually, the current Python code does consolidate all IOCs in to one threat report. I'm looking to have the ability to put IOCs in to different threat reports, within the same feed.

On Fri, Mar 8, 2019 at 2:15 PM Jason McFarland notifications@github.com wrote:

This is good feedback. We typically put each IOC in their own threat report because that is usually what is asked, but maybe allowing an option to consolidate all IOCs into one threat report is needed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/carbonblack/cbfeeds/issues/13#issuecomment-471043240, or mute the thread https://github.com/notifications/unsubscribe-auth/ASdNYjSeEI2X6Emr0DqnWxH6tx7xAe-8ks5vUrbRgaJpZM4JrBCT .

-- Tim Duckworth

[0duck0]

I just read back through bentedesco's comment. Personally, I'm not looking to concatenate reports, as he requested. I just want to have the ability to add one or more IOCs to a threat report and have that report added on to an existing feed. I have a "raw" feed setup locally but if I want to add to it, I have to recreate the feed in order to keep all previous IOCs as well as the new ones that I'm adding. I should be able to just add the additional IOCs as a new report.