Versioning search logic needs to be fixed

[himynamesdave]

Note, it is possible two identical objects are created, where the only difference is the _stix2arango_note property.

This is covered in arango_taxii_server_import_test_data.py (stix2arango utilities dir).

See example

curl -X 'GET' \
  'http://127.0.0.1:8000/api/taxii2/arango_taxii_server_tests_database/collections/mitre_attack_enterprise/objects/attack-pattern--2bce5b30-7014-4a5d-ade7-12913fe6ac36/?limit=10' \
  -H 'accept: application/taxii+json' \
  -H 'Authorization: Basic cmVhZF91c2VyOnRlc3RpbmcxMjM='

{
  "more": false,
  "next": null,
  "objects": [
    {
      "created": "2020-01-28T17:11:54.034Z",
      "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      "description": "Adversaries may clear system logs to hide evidence of an intrusion. macOS and Linux both keep track of system or user-initiated actions via system logs. The majority of native system logging is stored under the <code>/var/log/</code> directory. Subfolders in this directory categorize logs by their related functions, such as:(Citation: Linux Logs)\n\n* <code>/var/log/messages:</code>: General and system-related messages\n* <code>/var/log/secure</code> or <code>/var/log/auth.log</code>: Authentication logs\n* <code>/var/log/utmp</code> or <code>/var/log/wtmp</code>: Login records\n* <code>/var/log/kern.log</code>: Kernel logs\n* <code>/var/log/cron.log</code>: Crond logs\n* <code>/var/log/maillog</code>: Mail server logs\n* <code>/var/log/httpd/</code>: Web server access and error logs\n",
      "external_references": [
        {
          "source_name": "mitre-attack",
          "external_id": "T1070.002",
          "url": "https://attack.mitre.org/techniques/T1070/002"
        },
        {
          "source_name": "Linux Logs",
          "url": "https://www.eurovps.com/blog/important-linux-log-files-you-must-be-monitoring/",
          "description": "Marcel. (2018, April 19). 12 Critical Linux Log Files You Must be Monitoring. Retrieved March 29, 2020."
        }
      ],
      "id": "attack-pattern--2bce5b30-7014-4a5d-ade7-12913fe6ac36",
      "kill_chain_phases": [
        {
          "kill_chain_name": "mitre-attack",
          "phase_name": "defense-evasion"
        }
      ],
      "modified": "2021-04-29T14:49:39.188Z",
      "name": "Clear Linux or Mac System Logs",
      "object_marking_refs": [
        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
      ],
      "spec_version": "2.1",
      "type": "attack-pattern",
      "x_mitre_attack_spec_version": "2.1.0",
      "x_mitre_data_sources": [
        "File: File Deletion",
        "File: File Modification",
        "Command: Command Execution"
      ],
      "x_mitre_detection": "File system monitoring may be used to detect improper deletion or modification of indicator files. Also monitor for suspicious processes interacting with log files.",
      "x_mitre_domains": [
        "enterprise-attack"
      ],
      "x_mitre_is_subtechnique": true,
      "x_mitre_modified_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      "x_mitre_platforms": [
        "Linux",
        "macOS"
      ],
      "x_mitre_version": "1.0"
    },
    {
      "created": "2020-01-28T17:11:54.034Z",
      "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      "description": "Adversaries may clear system logs to hide evidence of an intrusion. macOS and Linux both keep track of system or user-initiated actions via system logs. The majority of native system logging is stored under the <code>/var/log/</code> directory. Subfolders in this directory categorize logs by their related functions, such as:(Citation: Linux Logs)\n\n* <code>/var/log/messages:</code>: General and system-related messages\n* <code>/var/log/secure</code> or <code>/var/log/auth.log</code>: Authentication logs\n* <code>/var/log/utmp</code> or <code>/var/log/wtmp</code>: Login records\n* <code>/var/log/kern.log</code>: Kernel logs\n* <code>/var/log/cron.log</code>: Crond logs\n* <code>/var/log/maillog</code>: Mail server logs\n* <code>/var/log/httpd/</code>: Web server access and error logs\n",
      "external_references": [
        {
          "source_name": "mitre-attack",
          "external_id": "T1070.002",
          "url": "https://attack.mitre.org/techniques/T1070/002"
        },
        {
          "source_name": "Linux Logs",
          "url": "https://www.eurovps.com/blog/important-linux-log-files-you-must-be-monitoring/",
          "description": "Marcel. (2018, April 19). 12 Critical Linux Log Files You Must be Monitoring. Retrieved March 29, 2020."
        }
      ],
      "id": "attack-pattern--2bce5b30-7014-4a5d-ade7-12913fe6ac36",
      "kill_chain_phases": [
        {
          "kill_chain_name": "mitre-attack",
          "phase_name": "defense-evasion"
        }
      ],
      "modified": "2021-04-29T14:49:39.188Z",
      "name": "Clear Linux or Mac System Logs",
      "object_marking_refs": [
        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
      ],
      "spec_version": "2.1",
      "type": "attack-pattern",
      "x_mitre_attack_spec_version": "2.1.0",
      "x_mitre_data_sources": [
        "File: File Deletion",
        "File: File Modification",
        "Command: Command Execution"
      ],
      "x_mitre_detection": "File system monitoring may be used to detect improper deletion or modification of indicator files. Also monitor for suspicious processes interacting with log files.",
      "x_mitre_domains": [
        "enterprise-attack"
      ],
      "x_mitre_is_subtechnique": true,
      "x_mitre_modified_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      "x_mitre_platforms": [
        "Linux",
        "macOS"
      ],
      "x_mitre_version": "1.0"
    },
    {
      "created": "2020-01-28T17:11:54.034Z",
      "created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      "description": "Adversaries may clear system logs to hide evidence of an intrusion. macOS and Linux both keep track of system or user-initiated actions via system logs. The majority of native system logging is stored under the <code>/var/log/</code> directory. Subfolders in this directory categorize logs by their related functions, such as:(Citation: Linux Logs)\n\n* <code>/var/log/messages:</code>: General and system-related messages\n* <code>/var/log/secure</code> or <code>/var/log/auth.log</code>: Authentication logs\n* <code>/var/log/utmp</code> or <code>/var/log/wtmp</code>: Login records\n* <code>/var/log/kern.log</code>: Kernel logs\n* <code>/var/log/cron.log</code>: Crond logs\n* <code>/var/log/maillog</code>: Mail server logs\n* <code>/var/log/httpd/</code>: Web server access and error logs\n",
      "external_references": [
        {
          "source_name": "mitre-attack",
          "external_id": "T1070.002",
          "url": "https://attack.mitre.org/techniques/T1070/002"
        },
        {
          "source_name": "Linux Logs",
          "url": "https://www.eurovps.com/blog/important-linux-log-files-you-must-be-monitoring/",
          "description": "Marcel. (2018, April 19). 12 Critical Linux Log Files You Must be Monitoring. Retrieved March 29, 2020."
        }
      ],
      "id": "attack-pattern--2bce5b30-7014-4a5d-ade7-12913fe6ac36",
      "kill_chain_phases": [
        {
          "kill_chain_name": "mitre-attack",
          "phase_name": "defense-evasion"
        }
      ],
      "modified": "2021-04-29T14:49:39.188Z",
      "name": "Clear Linux or Mac System Logs",
      "object_marking_refs": [
        "marking-definition--fa42a846-8d90-4e51-bc29-71d5b4802168"
      ],
      "spec_version": "2.1",
      "type": "attack-pattern",
      "x_mitre_attack_spec_version": "2.1.0",
      "x_mitre_data_sources": [
        "File: File Deletion",
        "File: File Modification",
        "Command: Command Execution"
      ],
      "x_mitre_detection": "File system monitoring may be used to detect improper deletion or modification of indicator files. Also monitor for suspicious processes interacting with log files.",
      "x_mitre_domains": [
        "enterprise-attack"
      ],
      "x_mitre_is_subtechnique": true,
      "x_mitre_modified_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5",
      "x_mitre_platforms": [
        "Linux",
        "macOS"
      ],
      "x_mitre_version": "1.0"
    }
  ]
}

3 objects are returned, all are identical

You can see this clearly in the versions endpoint

curl -X 'GET' \
  'http://127.0.0.1:8000/api/taxii2/arango_taxii_server_tests_database/collections/mitre_attack_enterprise/objects/attack-pattern--2bce5b30-7014-4a5d-ade7-12913fe6ac36/versions/?limit=10' \
  -H 'accept: application/taxii+json' \
  -H 'Authorization: Basic cmVhZF91c2VyOnRlc3RpbmcxMjM='

{
  "more": false,
  "next": null,
  "versions": [
    "2021-04-29T14:49:39.188Z",
    "2021-04-29T14:49:39.188Z",
    "2021-04-29T14:49:39.188Z"
  ]
}

Here, the expected result is only one objects is returned from both endpoint.

We need to set logic to determine unique "versions" of objects in TAXII, which is different to stix2arango.

This is somewhat solved for many endpoint where _is_latest==true is passed (#13 ) .

However, problems happen when match[version] parameter and versions endpoints are triggered (where object modified times are used).

We should account for this by only showing distinct results by id and modified time (the STIX spec states that if these two values are the same, the object should be identical. Put another way, if any value in object changed, modified time should also change)

As such the Arango search logic for this can be simply be to return distinct objects by id and modified time.

Thus for the above example, the versions shown would be

{
  "more": false,
  "next": null,
  "versions": [
    "2021-04-29T14:49:39.188Z"
  ]
}

For a test case with multiple versions you can use attack-pattern--67720091-eee3-4d2d-ae16-8264567f6f5b