File SCO with WindowsPEBinary using WindowsPESection causes JSON serialization error

[emmanvg]

Below an example of how to trigger the problem:

from stix2.v21 import observables

directory_observable = observables.Directory(path="/home/user-1/")
file_observable = observables.File(
    name="example.exe",
    parent_directory_ref=directory_observable["id"],
    size=68 * 1000,
    magic_number_hex="50000000",
    hashes={
        "SHA-256": "841a8921140aba50671ebb0770fecc4ee308c4952cfeff8de154ab14eeef4649"
    },
    extensions={"windows-pebinary-ext": observables.WindowsPEBinaryExt(pe_type="exe", machine_hex="014c", sections=[observables.WindowsPESection(name=".data", size=4096, entropy=7.980693, hashes={"SHA-256": "6e3b6f3978e5cd96ba7abee35c24e867b7e64072e2ecb22d0ee7a6e6af6894d0"})])}
)
print(directory_observable, file_observable)

I obtain the following error:

Traceback (most recent call last):
  File "C:/Users/user-1/home/example_project/exe_report.py", line 69, in <module>
    exe_report()
  File "C:/Users/user-1/home/example_project/exe_report.py", line 45, in exe_report
    extensions={"windows-pebinary-ext": observables.WindowsPEBinaryExt(pe_type="exe", machine_hex="014c", sections=[observables.WindowsPESection(name=".data", size=4096, entropy=7.980693, hashes={"SHA-256": "6e3b6f3978e5cd96ba7abee35c24e867b7e64072e2ecb22d0ee7a6e6af6894d0"})])}
  File "C:\Users\user-1\venv\lib\site-packages\stix2\base.py", line 317, in __init__
    possible_id = self._generate_id(kwargs)
  File "C:\Users\user-1\venv\lib\site-packages\stix2\base.py", line 395, in _generate_id
    data = canonicalize(streamlined_obj_vals, utf8=False)
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 502, in canonicalize
    textVal = JSONEncoder(sort_keys=True).encode(obj)
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 233, in encode
    chunks = list(chunks)
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 478, in _iterencode
    for thing in _iterencode_list(o, _current_indent_level):
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 367, in _iterencode_list
    for chunk in chunks:
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 450, in _iterencode_dict
    for chunk in chunks:
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 450, in _iterencode_dict
    for chunk in chunks:
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 367, in _iterencode_list
    for chunk in chunks:
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 491, in _iterencode
    o = _default(o)
  File "C:\Users\user-1\venv\lib\site-packages\stix2\canonicalization\Canonicalize.py", line 211, in default
    o.__class__.__name__,
TypeError: Object of type 'WindowsPESection' is not JSON serializable

[emmanvg]

The error seems to be related to logic on the Deterministic ID calculation of the object.

[chisholm]

I had a peek at this and I think I found the bug. The _generate_id() function makes calls to _recursive_stix_to_dict() to deep-convert the kwargs to a pure dict. That means that _STIXBase objects should never have been passed to the canonicalizer. So that conversion is failing.

The bug is that _recursive_stix_to_dict() doesn't work. First, you can see that it will break out of the loop when the first non-dict/_STIXBase valued property is found. That will cause it to miss things. Secondly, it doesn't handle lists. The windows-pebinary-ext extension has a list-valued property, sections. One of the _STIXBase objects lies inside the list, and the deep-converter needs to handle that.