Struggling to get dedup to work + missing documentation

nycalex commented 3 years ago

Hi, Looks like there's a mismatch between documentation and the app...

The user guide has no details about use of "JIRA dedup content" field that i see in the app. https://ta-jira-service-desk-simple-addon.readthedocs.io/en/stable/userguide.html#jira-dedup-behavior

I've been struggling to get dedup to work and i feel i'm not using it right.

For example, my description for vulnerability ticket is:

`Vulnerability name: $result.plugin_name$

Severity: $result.Severity_Level$

Description: $result.description$

Hostnames impacted: $result.hostname$

CVE involved: $result.cve$

`

My intention is to have splunk just update the existing ticket, so in the JIRA dedup content i specify $result.cve$. The idea is that for the given CVE, i would just keep getting jira ticket updates with new hosnames involved. But I'm still getting new jira issues opened instead.

I'll greatly appreciate any help.

guilhemmarchand commented 3 years ago

Hi @nycalex

Right, I agree the documentation could be a bit more complete on this feature especially, which use case can be more or less complex.

Thus, when you say a mismatch did you mean something in particular?

Regarding, the dedup, this is in the end simple enough:

Once you enable the dedup on a given alert, the TA calculates an MD5 hash for the full JSON payload that represents the incident send to JIRA.

This MD5 is registered in a KVstore, which is compared everytime a new JIRA issue is about to be created AND if the option is enabled.

Tip:

--> Enable the DEBUG logging --> In the TA logs, you will get more information about the reason why the TA made the decision or not, and especially you will see the entire JSON payload that is the final result sent to the API

The third option "JIRA dedup content" allows you to control with more granularity what you want to be taken into account during the MD5 calculation, without removing content from the result alert.

To say it differently, by default we take all the JSON payload and get the MD5, shall anything change in the JSON (like a different time) this leads to a new MD5, therefore to a new ticket.

If you need to restrict, perhaps to your CSV, then you would use the third option and restrict to this specific content, if the alert triggers, the MD5 is calculated against this content, and the associated issue would get a comment added instead of a new ticket.

Questions:

Can you share what your alert is looking like?
Can you enable the DEBUG mode, run a few attempts and verify the messages in the logs, as well as a quick comparison of the JSON?

I assume you may have always a difference in the JSON resulting which is why the TA creates a new issue.

Guilhem

guilhemmarchand commented 3 years ago

Any update @nycalex ?

nycalex commented 3 years ago

i do have the logs set to debug but i'm not sure which index/source the debugs are at.

about my alert to jira. i am using several variables to populate the content of the ticket. Value of some variables does not change over time the same, others vars change between the times when alert is scheduled to run.

For example:

Jira ticket name is based on the var that never changes
Body of the ticket is populated by some static text and 2 variables, $result.plugin_name$ and $result.hostname$

The $result.plugin_name$ variable is always the same, while $result.hostname$ changes.

To exclude ever-changing $result.hostname$ from being used as an input for md5 hash, i entered $result.plugin_name$ in the Jira Dedup Content field.

Is it how that field meant to be used?

I do see few examples there but i can't figure out what they mean... Listing just a var to be used for md5 makes sense, but what is the intended use for notations like : message=$result.message$

Thanks again

guilhemmarchand commented 3 years ago

@nycalex

First, logs are easily available via direct short cuts in the application UI, look at the navigation bar, this is clearly documented here:

https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/troubleshoot.html#overview-dashboard-and-add-on-logs

When you are in DEBUG mode, the final REST call, which in fact is a JSON structure, is logged in the Addon logs, search for:

json data for final rest call

Let's take the following simplistic example, I am using the following search to simulate a condition triggering my alert, which fires a JIRA alert action:

| makeresults
| eval user="foo@splunk.com", action="failure", reason="Authentication failed"
| eval time=strftime(_time, "%c")

This results in the following example, with a value of time that is different everytime the alert fires:

In my alert action, I do not for now select anything in the dedup option, so it's disabled by default, in the description I am adding various fields in the description including my time field which is going to be different all the time:

Looking at the logs, a fancy and perfectly formated JSON appears:

Several other information are provided, such MD5 information which are stored in a lookup:

2021-06-25 20:33:05,394 DEBUG pid=5759 tid=MainThread file=cim_actions.py:message:243 | sendmodaction - signature="jira_dedup: The calculated md5 hash for this issue creation request (db05a46bd3a2e6ccb57906cd749db047) was not found in the backlog collection, a new issue will be created" action_name="jira_service_desk" search_name="Test JIRA - demo dedup" sid="scheduler__admin__search__RMD526ad4cfa87997743_at_1624653180_13" rid="0" app="search" user="admin" action_mode="saved"

The MD5 is calculated against the entire JSON data, as explained via the logs and docs.

To access the KVstore collection containing these records, look at the nav menu "KVstore collections / JIRA Service Desk - Issues backlog collection".

As every ticket corresponds to a new issue, the status is "created".

Now, let's modify a bit the alert, I will remove the time field from the description in JIRA, and enable the dedup:

As the content of the JSON, of the issue, is exactly the same (I removed the time from the description), the Addon will detect it and perform an update of the issue, adding a comment, and updating the record in the KVstore lookup:

2021-06-25 20:45:06,360 INFO pid=8814 tid=MainThread file=cim_actions.py:message:243 | sendmodaction - signature="jira_dedup: An issue with same md5 hash (60727858c049e599fdb68a3cd744a911) was found in the backlog collection, as jira_dedup is enabled a new comment will be added if the issue is active. (status is not resolved or any other done status), entry:={ "jira_md5" : "60727858c049e599fdb68a3cd744a911", "ctime" : "1624652826.254012", "mtime" : "1624652826.2540202", "status" : "created", "jira_id" : "10100", "jira_key" : "LAB-76", "jira_self" : "https://localhost:8081/rest/api/2/issue/10100", "_user" : "nobody", "_key" : "60727858c049e599fdb68a3cd744a911" }" action_name="jira_service_desk" search_name="Test JIRA - demo dedup" sid="scheduler__admin__search__RMD526ad4cfa87997743_at_1624653900_33" rid="0" app="search" user="admin" action_mode="saved" action_status="success"

In the records:

key | ctime | mtime | jira_id | jira_key | jira_md5 | jira_self | status
-- | -- | -- | -- | -- | -- | -- | --
60727858c049e599fdb68a3cd744a911 | Fri Jun 25 20:27:06 2021 | Fri Jun 25 20:47:06 2021 | 10100 | LAB-76 | 60727858c049e599fdb68a3cd744a911 | https://localhost:8081/rest/api/2/issue/10100 | updated

Now, say I need to have the time information in the JIRA issue description content, but I do not want to create a new issue if the alert triggers for the same "condition", part of the logic can potentially handled in the alert itself (via the throttling etc) but from JIRA point of view, you may want to add a new comment instead of creating a new issue, so you need the alert to be triggering.

I can modify my alert, to create in Splunk a field which will be used for the MD5 calculation, based in this we will decide if a new issue should be created or a comment added:

| makeresults
| eval user="foo@splunk.com", action="failure", reason="Authentication failed"
| eval time=strftime(_time, "%c")
| eval dedup_condition = "user=" . user . "|action=" . action . "|reason=" . reason

Resulting in:

This is my condition, this can be simple like here but you could go very fancy too and do advanced decission automatically.

My alert action now looks like: (note I re-introduced the time field)

This answers one of your questions, once you the alert is setup and fires, instead of taking the full JSON for the calculation, we will use the content of the field represented by the token:

$result.dedup_condition

When the alert triggers, the JSON contains alll the information we need, including the field value changing at every trigger:

2021-06-25 21:09:05,025 DEBUG pid=15029 tid=MainThread file=cim_actions.py:message:243 | sendmodaction - signature="json data for final rest call:={
    "fields": {
        "project": {
            "key": "LAB"
        },
        "summary": "Splunk Alert: Test JIRA - demo dedup",
        "description": "The alert condition for 'Test JIRA - demo dedup' was triggered.\n\n- user: foo@splunk.com\n- action: failure\n- reason: Authentication failed\n- time: Fri Jun 25 21:09:00 2021",
        "issuetype": {
            "name": "Bug"
        },
        "priority": {
            "name": "High"
        }
    }
}" action_name="jira_service_desk" search_name="Test JIRA - demo dedup" sid="scheduler__admin__search__RMD526ad4cfa87997743_at_1624655340_75" rid="0" app="search" user="admin" action_mode="saved"

However, because we restrict what we use for the MD5 calculation, the MD5 remains the same therefore it is a duplicate effectively, we add a comment and update the records:

2021-06-25 21:09:05,032 INFO pid=15029 tid=MainThread file=cim_actions.py:message:243 | sendmodaction - signature="jira_dedup: An issue with same md5 hash (84381893e99c5abba0ad786969515f75) was found in the backlog collection, as jira_dedup is enabled a new comment will be added if the issue is active. (status is not resolved or any other done status), entry:={ "jira_md5" : "84381893e99c5abba0ad786969515f75", "ctime" : "1624655285.20085", "mtime" : "1624655285.200863", "status" : "created", "jira_id" : "10124", "jira_key" : "LAB-100", "jira_self" : "https://localhost:8081/rest/api/2/issue/10124", "_user" : "nobody", "_key" : "84381893e99c5abba0ad786969515f75" }" action_name="jira_service_desk" search_name="Test JIRA - demo dedup" sid="scheduler__admin__search__RMD526ad4cfa87997743_at_1624655340_75" rid="0" app="search" user="admin" action_mode="saved" action_status="success"

That's it! You are in full control, you can handle simple and very complex scenarios easily!

All good?

I will update the documentation to try going step by step the same way ;-)

Let me know

Guilhem

nycalex commented 3 years ago

Great, thanks so much!

guilhemmarchand commented 3 years ago

@nycalex

I have reviewed and updated the Addon docs to reflect these discussions, I included as well explanations about a few topics we haven't discusses such as controlling the content of the comment, and the JIRA issues closure.

https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/userguide.html#jira-dedup-behavior

guilhemmarchand / TA-jira-service-desk-simple-addon

Struggling to get dedup to work + missing documentation #88