search

crits / crits_services

CRITs Services Collection
182 stars 130 forks source link

ThreatExchange (nonexisting field (submitter_count)) #209

Closed thelok closed 8 years ago

thelok commented 8 years ago

I understand ThreatExchange is still under development, but just want to document this here.

Queried for malware families => variants => dropped by => Import

'status_code': 400, 'message': u'(#100) Tried accessing nonexisting field (submitter_count) on node type (MalwareAnalysis)', 'type': u'OAuthException'
mgoffin commented 8 years ago

Interesting. If you view that specific Malware Analyses via the web I assume the field doesn't exist either?

thelok commented 8 years ago

How do you view it via the web?

mgoffin commented 8 years ago

https://graph.facebook.com/[id]/?access_token=foo|bar

thelok commented 8 years ago

Man, getting the ID was really difficult. But the answer is no it doesn't exist in the web JSON result either.

thelok commented 8 years ago

There's a victim count though.

mgoffin commented 8 years ago

What if you add "&metadata=1" to the URL. does it show submitter_count as a viable field?

thelok commented 8 years ago

A whole bunch of extra fields but no submitter count

thelok commented 8 years ago

On a separate note, in the query builder the "Status" field is never used when submitting the form, probably because it has an id="text".

thelok commented 8 years ago

And "importing" IP_ADDRESS types don't work because the CRITs vocab doesn't define a IP_ADDRESS type =) Sounds like we'll need a mapping or a new type.

mgoffin commented 8 years ago

Won't need a new type. We purposely decided to specify whether something is IPv4 or IPv6. Same goes with exporting. It will require you to pick IP_ADDRESS at this time. But we will need to map on import I guess.

mgoffin commented 8 years ago

Added a couple things. Hopefully that maps at least the IP_ADDRESS and IP_SUBNET types to IPV4_ADDRESS and IPV4_SUBNET (we really should check to see if we should use v4 or v6). Also removed SUBMITTER_COUNT from something we query for on import since we don't really use it (still following up with the TX folks as I was unaware it was being removed). Also fixed the Status ID so it's actually "status" now. Let me know if those fixes help anything!

mgoffin commented 8 years ago

Ok after discussing with the ThreatExchange folks it seems submitter_count was removed from the Graph for all of the ThreatExchange objects. I updated pytx to 0.4.1 to support this and also updated the CRITs code again to remove it. You'll need to update to master and to the latest version of pytx.

thelok commented 8 years ago

That was fast, thanks!