mitre / cti

Cyber Threat Intelligence Repository expressed in STIX 2.0
Other
1.71k stars 410 forks source link

Broken CTI TAXII collection URL for attack mapping #190

Closed ag-michael closed 2 years ago

ag-michael commented 2 years ago

I've been using your STIX collections at https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/ , which has been working great until today. If you run this:

curl 'https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match\[type\]=attack-pattern' -H 'Accept: application/vnd.oasis.stix+json; version=2.0'

You will see the data being cut off mid-way, causing a JSON parsing error on clients.

ag-michael commented 2 years ago

For example, this is breaking attack scripts such as https://github.com/mitre-attack/attack-scripts/blob/b94e05c0a29a6fdfc9701c721fcf03cdf9f7945b/scripts/techniques_from_data_source.py

 python3 techniques_from_data_source.py                                 
All data sources in Enterprise ATT&CK:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/taxii2client/common.py", line 125, in _to_json
    return resp.json()
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Unterminated string starting at: line 1 column 64468 (char 64467)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mgetachew/attack-scripts/scripts/techniques_from_data_source.py", line 51, in <module>
    print("\n".join(data_sources()))
  File "/Users/mgetachew/attack-scripts/scripts/techniques_from_data_source.py", line 16, in data_sources
    techniques = tc_src.query([Filter("type", "=", "attack-pattern")])
  File "/usr/local/lib/python3.9/site-packages/stix2/datastore/taxii.py", line 290, in query
    all_data = self.collection.get_objects(**taxii_filters_dict).get('objects', [])
  File "/usr/local/lib/python3.9/site-packages/taxii2client/v20/__init__.py", line 419, in get_objects
    response = self._conn.get(self.objects_url, headers=headers, params=query_params)
  File "/usr/local/lib/python3.9/site-packages/taxii2client/common.py", line 325, in get
    return _to_json(resp)
  File "/usr/local/lib/python3.9/site-packages/taxii2client/common.py", line 128, in _to_json
    six.raise_from(InvalidJSONError(
  File "<string>", line 3, in raise_from
taxii2client.exceptions.InvalidJSONError: Invalid JSON was received from https://cti-taxii.mitre.org/stix/collections/95ecc380-afe9-11e4-9b6c-751b66dd541e/objects/?match%5Btype%5D=attack-pattern
ag-michael commented 2 years ago

This has been resolved.