For embedded relationship STIX SROs `created` and `modified` properties are not being assigned correctly

himynamesdave commented 1 week ago

This appears to have occurred since: https://github.com/muchdogesec/stix2arango/pull/14

However, this cannot be 100% validated.

Running

python3 -m unittest tests/test_18-testing-nested-embedded-ref.py

See the modified and created times in _is_ref objects is null

FOR doc IN test18_edge_collection
                FILTER doc._is_ref == true
                AND doc._is_latest == true
                LET keys = ATTRIBUTES(doc)
                LET filteredKeys = keys[* FILTER !STARTS_WITH(CURRENT, "_")]
                RETURN KEEP(doc, filteredKeys)

[
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--54d79501-6515-581f-b1a4-001d12c07560",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "address",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "cryptocurrency-wallet--d3efcf4e-d9fa-578d-a83f-f11af4c2504c",
    "type": "relationship"
  },
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--bec337cc-0433-53a8-bd35-c44e1db4f3c4",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "address",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "cryptocurrency-wallet--f17481a9-a6ad-5ebc-af2a-9b637dab774b",
    "type": "relationship"
  },
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--5d606a3b-7b13-5404-8ce2-59438323a64a",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "address",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "cryptocurrency-wallet--9edba811-a14c-5795-8dd6-e39b4bde1c40",
    "type": "relationship"
  },
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--5c52661b-b0ea-5387-907f-eae2e6c0d548",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "address",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "cryptocurrency-wallet--7052df7c-e9f4-5fc9-bc57-304278647176",
    "type": "relationship"
  },
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--35f9c60e-5364-556e-a6f0-ccb0179eec02",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "object-marking",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
    "type": "relationship"
  },
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--aba6ef16-1d33-5e08-bcb6-86c0696071f0",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "object-marking",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
    "type": "relationship"
  },
  {
    "created": null,
    "created_by_ref": "identity--72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4",
    "id": "relationship--b1faec5b-d89c-58aa-a2d6-0b41414fc47f",
    "modified": null,
    "object_marking_refs": [
      "marking-definition--904ac99b-7539-5de7-9ffa-23186f0e07b6",
      "marking-definition--27557362-b745-4161-96e8-ccd62ce4cb26",
      "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
    ],
    "relationship_type": "object-marking",
    "source_ref": "cryptocurrency-transaction--f437c493-b651-5cbb-845a-3dd231a39ec6",
    "spec_version": "2.1",
    "target_ref": "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487",
    "type": "relationship"
  }
]

EXPECTED

Expected output is

{
    "type": "relationship",
    "spec_version": "2.1",
    "id": "relationship--<UUID v5>",
    "created_by_ref": "<IMPORTED STIX2ARANGO IDENTITY ID>",
    "relationship_type": "<EMBEDDED RELATIONSHIP TYPE WITHOUT REF OR REFS AND _ REPLACED WITH ->",
    "created": "<OBJECT CREATED TIME>",
    "modified": "<OBJECT MODIFIED TIME>",
    "source_ref": "<OBJECT WITH REF PROPERTY>",
    "target_ref": "<REF TARGET PROPERTY>",
    "object_marking_refs": [
        "<object_marking_refs SHOULD MATCH SOURCE_REF OBJECT>"
    ]
}

Note, created is first time this relationship id was generated, and should never be changed

On first creation, created and modified times are the same. If any updates to this embedded relationship happen (id) then the new object has the same created time, but an increased modified time.

ANALYSIS

Currently embedded SROs try to hook into the created and modified time of the source_ref object

This approach causes failures b/c SROs (e.g. ipv4-addr, crytocurrency-wallet, etc) do not have created and modified properties.

Hence you need to use the server creation time, but also have the logic to keep the created time for the id persistent when updates happen to original objects that contain the embedded refs these properties belong to.

TESTS

I've added some tests for this

https://github.com/muchdogesec/stix2arango/commit/e2088134e5a21651d04a34b3809d27119e3be987

himynamesdave commented 6 days ago

@fqrious I noticed the docs for this were wrong. I have updated to cover this case.

fqrious commented 1 day ago

This is kinda hard to implement, what do you think about using a default created time and a modified time instead? if we have the default creation time as 2020-01-01, there won't be a need to query the database for existing date first.

created: 2020-01-01
modified: <current-time>

himynamesdave commented 1 day ago

@fqrious we won't be able to get away with that.

created time is important.

If this logic is complex, can't we just implement the same logic that already exists for updating SDOs (that is other objects with created and modified times? That is very similar to this approach, and would achieve the same thing?

fqrious commented 1 day ago

No, existing SDOs come with their own created time, we don't need to probe for old upload's created time for anything

himynamesdave commented 1 day ago

yea, my point being, why don't we just do this here (for relationships that is)?

That way we don't need to probe modified time, and the behaviour across all objects with modified and created properties is the same?

This is different to the spec above, but seems like a good solution, if you think so too?

himynamesdave commented 1 day ago

Currently embedded relationship objects created by stix2arango get their created and modified times from the source object.

This is the cause of this issue, because not all src objects have this time.

So this means for any embedded relationships being generated with a stix src object with no modified or created times, the embedded relationship objects generated for them should NOT have either a created or modified time.

Note, some src object only contain a created time. In this case, the relationships created from it should inherit a created time, but no modified time

In short, any embedded relationships created by stix2arango should have created and modified properties that match the source object they were created from. If they don't exist, don't include them

fqrious commented 1 day ago

Oh, you're saying it shouldn't be null... The key should just not be on the object?

That it?

himynamesdave commented 21 hours ago

Yes, as simple as that

muchdogesec / stix2arango