Pattern objects cannot be converted to JSON

[ghost]

Input is a Stix 1.2 XML document with an structure like:

report -> indicator -> observable

The observable is converted to a stix2elevator.convert_pattern.ComparisonExpressionForElevator but when the stix2 JSON output is attempted to be written, I get an error like:

TypeError: <stix2elevator.convert_pattern.ComparisonExpressionForElevator object at 0x7f019b965fd0> is not JSON serializable

[ghost]

Note that if I nest the same observable under the following structure:

reports -> observables -> observable

Then I don't get an error and the output looks OK, although the link between the incident and the observable no longer exists.

[ghost]

To add even more detail, it appears a completely different function is called to convert Observables depending on whether they are nested under a Report or an Indicator:

== Under an indicator (convert_stix.py)
if indicator.observable is not None:
    indicator_instance["pattern"] = convert_observable_to_pattern(indicator.observable)

== Under a report (convert_stix.py)
if report.observables:
    for o_d in report.observables:
        if o_d.id_ is not None:
            o_d20 = convert_observed_data(o_d, bundle_instance, parent_created_by_ref, parent_timestamp)

These two functions return totally different objects, and only one can be converted to JSON. Why is the handling of the observable so different in the location of the Observable?

[gtback]

Hi @fcntl, thanks for reporting this.

The primary reason for the current behavior is that STIX 1.x uses an "Observable" for both instances of cyber observable data, and also patterns to look for as part of indicators. STIX 2 makes a distinction between "Observed Data" (the former) and "Indicators" (the latter). The elevator makes a decision about which to create based on the location in the STIX 1 document.

Given all that, there's definitely still an error with serializing the patterns (STIX 2 indicators created from STIX 1.x observables within STIX 1.x indicators). We'll look into this.

[ghost]

@gtback thanks for the explanation.

[cyberfox1]

Hello, I also had this issue. It seems the problem is indicators are converted to objects in stix2, yet when finalizing the package the code looking for patterns iterates over indicators not objects, e.g. somewhere near line 1600 in convert_stix.py:

  if "indicators" in bundle_instance:
      interatively_resolve_placeholder_refs()
      for ind in bundle_instance["indicators"]:

If that last line in changed to:

      for ind in bundle_instance["objects"]:

I at least get a valid JSON output without the exception being thrown.

[gtback]

Thanks, @cyberfox1

[rpiazza]

@cyberfox1, that change won't help. The elevator was implemented early in the 2.0 specification process. Originally in STIX 2.0, a bundle had a property for each SDO/SRO type. This was dropped, and replaced by just the objects property, but much code was written to support the original properties - so in finalize_bundle, we collapse all of separate properties into the one objects properties. As we all know, implementing to a "moving" specification target sometimes introduces convoluted code...

[rpiazza]

@cyberfox1, do you have some example STIX content that I can use to investigate this issue? Thanks in advance!!

[cyberfox1]

Hi @rpiazza , I didn't think the change was the final solution, but it did get me past a blocking point where I can produce valid stix2 from stix1 using the elevator, for the following test document for example.

<stix:STIX_Package 
    xmlns:tlpMarking="http://data-marking.mitre.org/extensions/MarkingStructure#TLP-1"
    xmlns:stix="http://stix.mitre.org/stix-1"
    xmlns:AddressObj="http://cybox.mitre.org/objects#AddressObject-2"
    xmlns:ttp="http://stix.mitre.org/TTP-1"
    xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:et="http://stix.mitre.org/ExploitTarget-1"
    xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1"
    xmlns:cybox="http://cybox.mitre.org/cybox-2"
    xmlns:indicator="http://stix.mitre.org/Indicator-2"
    xmlns:marking="http://data-marking.mitre.org/Marking-1"
    xmlns:report="http://stix.mitre.org/Report-1"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:stixCommon="http://stix.mitre.org/common-1"
    xmlns:example="http://example.com"
    xmlns:cyboxCommon="http://cybox.mitre.org/common-2"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns:simpleMarking="http://data-marking.mitre.org/extensions/MarkingStructure#Simple-1"
    xmlns:DomainNameObj="http://cybox.mitre.org/objects#DomainNameObject-1"
     id="example:Package-3dee54a0-cd51-409c-93d1-5209e9c7299b" version="1.2">
    <stix:STIX_Header>
        <stix:Description>Doc Header</stix:Description>
    </stix:STIX_Header>
    <stix:Reports>
        <stix:Report timestamp="2017-12-18T04:14:38.069315Z" id="example:Report-e33e4af9-1397-4af6-86d7-ac1a12de7402" xsi:type='report:ReportType' version="1.0">
            <report:Header>
                <report:Description>Report Header</report:Description>
            </report:Header>
            <report:Indicators>
                <report:Indicator id="indicator--07c221ae-fe63-4841-970e-4e86d9c89408" timestamp="2017-05-30T01:11:51+00:00" xsi:type='indicator:IndicatorType'>
                    <indicator:Title>Indicator1</indicator:Title>
                    <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">Domain Watchlist</indicator:Type>
                    <indicator:Description>first</indicator:Description>
                    <indicator:Observable id="example:Observable-5096ae7a-d62f-4493-b350-12a93c06e316">
                        <cybox:Object id="example:DomainName-2629f1bc-f0ec-4512-9626-57266a5f80c6">
                            <cybox:Properties xsi:type="DomainNameObj:DomainNameObjectType">
                                <DomainNameObj:Value>www.yahoo.com</DomainNameObj:Value>
                            </cybox:Properties>
                        </cybox:Object>
                    </indicator:Observable>
                </report:Indicator>
            </report:Indicators>
        </stix:Report>
    </stix:Reports>
</stix:STIX_Package>

I can validate the above using stix-validator, and the ouput from the elevator is the following:

{
    "id": "bundle--3dee54a0-cd51-409c-93d1-5209e9c7299b",
    "objects": [
        {
            "created": "2017-05-30T01:11:51.000Z",
            "description": "first",
            "id": "indicator--07c221ae-fe63-4841-970e-4e86d9c89408",
            "labels": [
                "domain-watchlist"
            ],
            "modified": "2017-05-30T01:11:51.000Z",
            "name": "Indicator1",
            "pattern": "[domain-name:value = 'www.yahoo.com']",
            "type": "indicator",
            "valid_from": "2017-05-30T01:11:51.000000Z"
        },
        {
            "created": "2017-12-18T04:14:38.069Z",
            "description": "Report Header\n\nINTENT:\n",
            "id": "report--e33e4af9-1397-4af6-86d7-ac1a12de7402",
            "labels": [
                "unknown"
            ],
            "modified": "2017-12-18T04:14:38.069Z",
            "object_refs": [
                "indicator--07c221ae-fe63-4841-970e-4e86d9c89408"
            ],
            "type": "report"
        }
    ],
    "spec_version": "2.0",
    "type": "bundle"
}

[rpiazza]

Thanks for the test case @cyberfox1 !!

I understand, your "fix" was just to get some output - but I wanted to explain the somewhat convoluted code :-)

[cyberfox1]

@rpiazza no worries, I would have sent a pull request but it took long enough to find that "fix" and I wouldn't know where to begin finding the proper solution :)

[rpiazza]

@cyberfox1: We will be releasing this fix soon, but if you want to preview it you should look at commit 7a9633d (line 1165 of convert_stix.py)

[cyberfox1]

@rpiazza awesome, thanks!