[Filebeat][httpjson]Ability to persist state between events

[P1llus]

Creating an initial issue for discussion based on comment from @andrewkroh

When querying an API there is a need to persist state between each API call. Currently we do not have the possibility to store state in the registry file, so this issue is to track different user stories that would be great to support in the future.

User story: Azure ATP Allows querying with URL parameters using ODATA queries documented: https://docs.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/exposed-apis-odata-samples

When starting up a beat for the first time, it will retrieve all events, and for each consequent event it would be nice to be able to track a value from the last ingested event to be used as a query filter.

Initial request: https://api-eu.securitycenter.windows.com/api/alerts

Response:

{
    "@odata.context": "https://api-eu.securitycenter.windows.com/api/$metadata#Alerts",
    "value": [
        {
            "id": "da637249026486247330_1412635475",
            "incidentId": 2,
            "investigationId": null,
            "assignedTo": null,
            "severity": "Medium",
            "status": "New",
            "classification": null,
            "determination": null,
            "investigationState": "UnsupportedAlertType",
            "detectionSource": "WindowsDefenderAtp",
            "category": "InitialAccess",
            "threatFamilyName": null,
            "title": "Suspicious behavior by Microsoft Word was observed",
            "description": "A suspicious behavior by Microsoft Word application was observed. This behavior may indicate that a Word document was used to deliver malware or initiate other malicious activities on the machine.",
            "alertCreationTime": "2020-05-12T17:50:48.5153821Z",
            "firstEventTime": "2020-05-12T17:49:44.3584856Z",
            "lastEventTime": "2020-05-12T17:50:11.9889024Z",
            "lastUpdateTime": "2020-05-12T20:15:58.81Z",
            "resolvedTime": null,
            "machineId": "4d0ddab627ea0e54dc6ea75196b4b39db297f110",
            "computerDnsName": "testmachine1",
            "aadTenantId": "2978caa4-d66c-4c7e-9e30-40034eb7c6f3",
            "relatedUser": {
                "userName": "administrator1",
                "domainName": "TestMachine1"
            },
            "comments": [],
            "evidence": []
        },
        {
            "id": "da637249026573326737_-932580695",
            "incidentId": 1,
            "investigationId": null,
            "assignedTo": null,
            "severity": "Low",
            "status": "New",
            "classification": null,
            "determination": null,
            "investigationState": "InnerFailure",
            "detectionSource": "WindowsDefenderAv",
            "category": "Malware",
            "threatFamilyName": "Wintapp",
            "title": "'Wintapp' backdoor was detected",
            "description": "Backdoors are malicious remote access tools that allow attackers to access and control infected machines. Backdoors can also be used to exfiltrate data.\n\nThis detection might indicate that the malware was stopped from delivering its payload. However, it is prudent to check the machine for signs of infection.",
            "alertCreationTime": "2020-05-12T17:50:57.3326737Z",
            "firstEventTime": "2020-05-12T17:49:49.791639Z",
            "lastEventTime": "2020-05-12T17:49:49.791639Z",
            "lastUpdateTime": "2020-05-12T17:51:43.53Z",
            "resolvedTime": null,
            "machineId": "4d0ddab627ea0e54dc6ea75196b4b39db297f110",
            "computerDnsName": "testmachine1",
            "aadTenantId": "2978caa4-d66c-4c7e-9e30-40034eb7c6f3",
            "relatedUser": null,
            "comments": [],
            "evidence": []
        }
    ]
}

Any new API calls could use a field of choice by the user. The field type could also be a choice for the user between int and date (with a date format if needed).

We then need a way to define where the value is to be inserted, in this example it would be a URL parameter in the GET request:

Based on the newest incidentId of the newest event: https://api-eu.securitycenter.windows.com/api/alerts?$filter=incidentId+gt+2

Based on the newest alertCreationTime of the newest event: https://api-eu.securitycenter.windows.com/api/alerts?$filter=alertCreationTime+gt+2020-05-12T17:50:48.5153821Z

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

As of now I think pagination only supports setting HTTP headers, HTTP POST body parameters, or the full request URL. So we will need an enhancement (and possibly refactoring) to allow setting query parameters in the request URL.

[P1llus]

Just a quickly written example, I feel state/pagination should support:

• Set initial URL
• Set url to run after the initial url, maybe supporting a variable like "https://elastic.co/{%variable%}"
• Set which field in the JSON object to retrieve the newest value to track state from. Needs to support object Arrays or in a combination with json_objects_array which might be a better idea.
• Set dynamic variable type (Date with date format, or int/long).
• Set POST Body if not GET.
• POST Body should have a similar way to inject where the variable is needed.
• Still able to set/retrieve headers.

[marc-gr]

Closing this in favor of https://github.com/elastic/beats/issues/19486