Duplicate data detection

[Danko90]

Hi,

I read on the documentation that you are implementing duplicate data detection and I just wanted to know how long it takes to do this and eventually I can help with this.

Thanks

[FloatingGhost]

It's done. Pull and try that.

[Danko90]

@FloatingGhost , can you please tell me in which part of source code it's implemented?

[FloatingGhost]

Here ya dummy

[Danko90]

There's no need to insult by the way. I was asking because I pulled the FSISAC repository twice and I have two or more same events.

[Rafiot]

All ya silly native English speakers with your colloquialisms...

@Danko90 : I don't think @FloatingGhost means anything mean by it ;)

[davidljohnson]

Hey, don't murder me for bringing this up again, but I'm having a problem with duplicates and events with zero attributes. This is what my MISP instance looks like after running this a few days unattended to pull FS-ISAC data:

I updated this repo along with MISP, PyMISP, and MISP_STIX_Converter today and I'm still experiencing this problem. Here's the kind of logs I'm getting after running run-taxii-poll.py:

2017-06-30` 16:52:11,687 - main - DEBUG - Pushing block <cabby.entities.ContentBlock object at 0x7f929a152f98> 2017-06-30 16:52:11,777 - main - DEBUG - Pushing block <cabby.entities.ContentBlock object at 0x7f929a161358> 2017-06-30 16:52:11,864 - main - DEBUG - Pushing block <cabby.entities.ContentBlock object at 0x7f929a159828> 2017-06-30 16:52:11,949 - main - ERROR - FAILED TO PUSH BLOCK! 2017-06-30 16:52:11,950 - main - ERROR - <cabby.entities.ContentBlock object at 0x7f929a159828> 2017-06-30 16:52:11,950 - main - ERROR - FAILURE: There was a failure while executing the message handler Traceback (most recent call last): File "/var/git/MISP-Taxii-Server/scripts/run-taxii-poll.py", line 109, in uri=localInbox) File "/usr/local/lib/python3.5/dist-packages/cabby/client11.py", line 332, in push service_type=const.SVC_INBOX) File "/usr/local/lib/python3.5/dist-packages/cabby/abstract.py", line 205, in _execute_request timeout=self.timeout) File "/usr/local/lib/python3.5/dist-packages/cabby/dispatcher.py", line 91, in send_taxii_request raise UnsuccessfulStatusError(obj) cabby.exceptions.UnsuccessfulStatusError: FAILURE: There was a failure while executing the message handler 2017-06-30 16:52:11,951 - main - DEBUG - Pushing block <cabby.entities.ContentBlock object at 0x7f929a2bf978> 2017-06-30 16:52:12,033 - main - ERROR - FAILED TO PUSH BLOCK! 2017-06-30 16:52:12,033 - main - ERROR - <cabby.entities.ContentBlock object at 0x7f929a2bf978>

Any ideas, besides the obvious (abandoning STIX altogether)?

[adulau]

@obsidianpentesting I was trying to get an FS-ISAC feed for testing but without success until now. Do you know if you could share the feed with us? to make some tests.

[FloatingGhost]

I can't do much without the server log :P

The error will be in there

[davidljohnson]

@FloatingGhost Sorry for the wait. Had minimal computer access over the past few days. So the MISP server logs are interesting. It looks like some attributes are labeled incorrectly as "ip-src" when they should be email addresses:

Validation errors: {"value":["IP address has an invalid format."]} Full Attribute: {"value":"some_email@somedomain.com","comment":"Address : some_email@somedomain.com","to_ids":true,"disable_correlation":false,"category":"Network activity","type":"ip-src","distribution":"5","AttributeTag":[],"event_id":"32888"}

So this is the reason I'm seeing empty attributes for these FS-ISAC events. Does this need to be changed in MISP-STIX-Converter/misp_stix_converter/converters/buildMISPAttribute.py for data type validation?

Edit: Looks like this should probably be in a different thread. My bad!

[davidljohnson]

@adulau I can't give you direct access to the feed, but If I can find a way to obfuscate the IOCs (some are pretty revealing by themselves) and just keep the rest of the JSON output the same, I will share that output with you.