Integration and data not populated in splunk -jira-service desk-simple-addon

[cyberlearner95]

Hey i am working on getting the audit logs of jira to splunk integration , i found jira service desk simple addon and installed it on search head of my splunk. we configured the authentication using the basic authentication method consists of username(mail id) and api token(both created by site admin on jira side). i followed the config instruction as mentioned in this (https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/configuration.html) After configuration like shown in image below jira service desk addon showing only seven projects on jira projects overview tab but we have more than 50+ projects and jira issues overview tab and other options are not populated. we doesnot get any error in internal logs of splunk side to trouble shoot also. can you please help me out with solution

[guilhemmarchand]

Hi @cyberlearner95

Either this is a limitation of the map command running underneath, either a limitation of the account used for the JIRA connection which wouldn't be able to access to the other projects.

You can first list the projects running using the folowing report:

JIRA Service Desk - Get projects

Then, if you can see all the projects, try the following search within the JIRA app:

Replace "LAB" with the name of the account as you configured it in the app

What differs in this search from the one in the dashboard is the maxsearches argument, note that the LAB keyword is in several parts of the search. (the dashboard generates this search dynamically)

| makeresults | eval projects_csv="LAB" | makemv delim="," projects_csv | rename projects_csv as key | mvexpand key | streamstats count as project_no

| map [ | jirarest account="LAB" target="rest/api/2/search?jql=project=$key$&maxResults=0" ] maxsearches=100 | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_issues>\d*)"
| append [ 
| makeresults | eval projects_csv="LAB" | makemv delim="," projects_csv | rename projects_csv as key | mvexpand key | streamstats count as project_no
| map [ | jirarest account="LAB" target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22Done%22%29&maxResults=0" ] maxsearches=100 | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_done>\d*)"
]
| append [ 
| makeresults | eval projects_csv="LAB" | makemv delim="," projects_csv | rename projects_csv as key | mvexpand key | streamstats count as project_no
| map [ | jirarest account="LAB" target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22To%20Do%22%29&maxResults=0" ] maxsearches=100 | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_to_do>\d*)"
]
| append [ 
| makeresults | eval projects_csv="LAB" | makemv delim="," projects_csv | rename projects_csv as key | mvexpand key | streamstats count as project_no
| map [ | jirarest account="LAB" target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22In%20Progress%22%29&maxResults=0" ] maxsearches=100 | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_in_progress>\d*)"
]
| append [ | makeresults | eval projects_csv="LAB" | makemv delim="," projects_csv | rename projects_csv as key | mvexpand key | streamstats count as project_no ]
| eval line_merge=case(isnum(project_no), project_no, isnum(result_no), result_no)
| stats first(key) as project, first(total_*) as "total_*" by line_merge | fields - line_merge

| appendpipe [ stats sum(total_done) as total_done, sum(total_in_progress) as total_in_progress, sum(total_issues) as total_issues, sum(total_to_do) as total_to_do ]
| fillnull value="TOTAL" project

| eval pct_total_done="% " . round(total_done/total_issues*100, 2), pct_total_to_do="% " . round(total_to_do/total_issues*100, 2), pct_total_in_progress="% " . round(total_in_progress/total_issues*100, 2)
| foreach pct_* [ eval <<FIELD>> = if(isnull('<<FIELD>>'), "% 0.00", '<<FIELD>>' ) ]

| eval _time=now() | fields _time, project, pct_*, total_* | fields - _time | rename project as Project, pct_total_done as "% Done", pct_total_in_progress as "% In Progress", pct_total_to_do as "% To Do", total_done as "# Done", total_in_progress as "# In Progress", total_issues as "# Total", total_to_do as "# To Do"

Let me know.

[cyberlearner95]

Thank you so much for your quick response @guilhemmarchand (I have used admin credentials only for all the configuration) 1) I tried the first option you have mentioned in the comment by running the report JIRA Service Desk - Get projects and i am getting all the respective projects in the report. 2) I ran the query which you have given but this query not producing any project count. you can see in the attached image.

[cyberlearner95]

when i search with this query it is producing me all the projects please see below. image

[guilhemmarchand]

@cyberlearner95

Good. Can you please try the version of this dashboard, see attached, created a new dashboard in the app and copy paste the source code.

overview_jira_analytic.xml.txt

[cyberlearner95]

Tried with this code still same result only 6 projects are appearing on the dashboard.

[cyberlearner95]

Also jira service desk tab is not showing any result. please see below image ![Uploading jira test1.PNG…]()

[guilhemmarchand]

@cyberlearner95

Please test the following query:

| jirarest account=LAB target="rest/api/2/project" | spath | rename "{}.key" as key | table key | rename key as projects | eval projects_csv=mvjoin(projects, ",")

Replace "LAB" by the account name as it is configured in the Configuration account page (very first entry at the top), we expect as CSV list of all available JIRA projects.

Let me know

[cyberlearner95]

i tested the query only 6 projects i am getting in the result

[guilhemmarchand]

Also jira service desk tab is not showing any result. please see below image

I don't know what you mean, if you mean the default overview dashboard, it requires that:

• You have access to the Splunk _internal index

(index="_internal" OR index="cim_modactions") (source="*jira_service_desk_modalert.log")

• You have active alerts being created by the alert action

[guilhemmarchand]

i tested the query only 6 projects i am getting in the result

Ok so this is the root cause:

| jirarest account=LAB target="rest/api/2/project" | spath

Do you see all projects in the raw JSON?

[cyberlearner95]

please see below image

[cyberlearner95]

when i search with this query it is producing me all the projects please see below. image

i have already edited the query which you have provided and it is producing the all 65 projects you can see in the above image

[cyberlearner95]

i tested the query only 6 projects i am getting in the result

Ok so this is the root cause:

| jirarest account=LAB target="rest/api/2/project" | spath

Do you see all projects in the raw JSON?

No only 6 projects are appearing.

[guilhemmarchand]

Ok @cyberlearner95 so the fix is

spath output=projects path={}.key

1. Can you confirm the list is complete:

| jirarest account=LAB target="rest/api/2/project" | spath output=projects path={}.key | stats values(projects) as projects | eval projects_csv=mvjoin(projects, ",") | fields projects_csv

Please confirm it provides now the full list of projects as a CSV

[guilhemmarchand]

And please confirm the dashboard works with the map change (the maxsearch increased)

[cyberlearner95]

Ok @cyberlearner95 so the fix is

spath output=projects path={}.key

Can you confirm the list is complete:

| jirarest account=LAB target="rest/api/2/project" | spath output=projects path={}.key | stats values(projects) as projects | eval projects_csv=mvjoin(projects, ",") | fields projects_csv

Please confirm it provides now the full list of projects as a CSV

Now it is providing the all projects list ![Uploading jira test3.PNG…]()

[cyberlearner95]

should i update this query in the dashboard?

[guilhemmarchand]

Please verify and confirm this version of the dashboard:

overview_jira_analytic_v2.xml.txt

Once you confirm, I will later on publish a corrective release, you can either decide to:

• Wait for the new release
• Create a local copy of the dashboard and use it instead temporary until the ap is updated and the dashboard that comes with it is updated
• Update the embedded dashboard which creates a local copy of it, however for any future change the local copy will always override the default version shipped with the app, your choice and your responsability to handle and manage it

[cyberlearner95]

Please verify and confirm this version of the dashboard:

overview_jira_analytic_v2.xml.txt

Once you confirm, I will later on publish a corrective release, you can either decide to:

* Wait for the new release

* Create a local copy of the dashboard and use it instead temporary until the ap is updated and the dashboard that comes with it is updated

* Update the embedded dashboard which creates a local copy of it, however for any future change the local copy will always override the default version shipped with the app, your choice and your responsability to handle and manage it

should i update maxsearch value as well? I have updated the code locally but it is taking too much time to load the dashboard(almost 5 minutes still dashboard is loading )

[cyberlearner95]

Please verify and confirm this version of the dashboard: overview_jira_analytic_v2.xml.txt Once you confirm, I will later on publish a corrective release, you can either decide to:

* Wait for the new release

* Create a local copy of the dashboard and use it instead temporary until the ap is updated and the dashboard that comes with it is updated

* Update the embedded dashboard which creates a local copy of it, however for any future change the local copy will always override the default version shipped with the app, your choice and your responsability to handle and manage it

should i update maxsearch value as well? I have updated the code locally but it is taking too much time to load the dashboard(almost 5 minutes still dashboard is loading )

from last 10 min it is in loading state.

[cyberlearner95]

Data is not populating on dashboard with updated source code could you please check it again?

[guilhemmarchand]

I will review - you can try to remove the max search from the map command.

I will check the design and see how it can be improved.

[cyberlearner95]

I will review - you can try to remove the max search from the map command.

I will check the design and see how it can be improved.

i tried with this way but no luck, when i can expect this issue will be fix?

[guilhemmarchand]

@cyberlearner95

Yes, it will be fixed.

Can you try this version of the dashboard and let me know if this is working best without the maxsearch.

overview_jira_analytic_v3.xml.txt

The issue is basically the following, each metric to be retrieved needs to be specifying the project and there are about 4 metrics per project, so in the end the dashboard needs to 4 * number of projects REST calls to JIRA to retrieve all the informations for all the projects.

• One would would be either to find a way to have a single REST call to retrieve the KPI for all projects in one operation then having only 4 REST calls to be made.

• Another way to get you what you are after would be setup scheduled reports which runs collect to generate the summary information for the projects you care, as documented here:

https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/userguide.html#jira-rest-api-wrapper

Checkout the report:

JIRA Service Desk - Issues statistics report per project

Modify the spath, so it is:

| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no

| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_issues>\d*)"
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22Done%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_done>\d*)"
]
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22To%20Do%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_to_do>\d*)"
]
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22In%20Progress%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_in_progress>\d*)"
]
| append [ | jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no ]
| eval line_merge=case(isnum(project_no), project_no, isnum(result_no), result_no)
| stats first(key) as project, first(total_*) as "total_*" by line_merge | fields - line_merge

| eval pct_total_done=round(total_done/total_issues*100, 2), pct_total_to_do=round(total_to_do/total_issues*100, 2), pct_total_in_progress=round(total_in_progress/total_issues*100, 2)
| foreach pct_* [ eval <<FIELD>> = if(isnull('<<FIELD>>'), "0.00", '<<FIELD>>' ) ]

| where project!="noop"

| eval _time=now() | fields _time, project, pct_*, total_*

Does it run in an acceptable time frame? If so you could include a summary collect to index the results, then you do analytic on top of that with the historical vision too.

[cyberlearner95]

I tried with the updated code but same issue i am not getting overall projects. and if i remove maxsearch then also it is taking time to display the result on panel

[cyberlearner95]

I tried with the updated code but same issue i am not getting overall projects. and if i remove maxsearch then also it is taking time to display the result on panels almost it is taking 6 minutes to display the result

[guilhemmarchand]

So this:

| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no

| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_issues>\d*)"
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22Done%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_done>\d*)"
]
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22To%20Do%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_to_do>\d*)"
]
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22In%20Progress%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_in_progress>\d*)"
]
| append [ | jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no ]
| eval line_merge=case(isnum(project_no), project_no, isnum(result_no), result_no)
| stats first(key) as project, first(total_*) as "total_*" by line_merge | fields - line_merge

| eval pct_total_done=round(total_done/total_issues*100, 2), pct_total_to_do=round(total_to_do/total_issues*100, 2), pct_total_in_progress=round(total_in_progress/total_issues*100, 2)
| foreach pct_* [ eval <<FIELD>> = if(isnull('<<FIELD>>'), "0.00", '<<FIELD>>' ) ]

| where project!="noop"

| eval _time=now() | fields _time, project, pct_*, total_*

is taking a while and not performing properly right?

[cyberlearner95]

So this:

| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no

| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_issues>\d*)"
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22Done%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_done>\d*)"
]
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22To%20Do%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_to_do>\d*)"
]
| append [
| jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no
| map [ | jirarest account=_any target="rest/api/2/search?jql=project=$key$%20AND%20statuscategory%20IN%20%28%22In%20Progress%22%29&maxResults=0" ] | streamstats count as result_no | rex field=_raw "\"total\":\s(?<total_in_progress>\d*)"
]
| append [ | jirarest account=_any target="rest/api/2/project" | spath output=key path={}.key | table key | mvexpand key | append [ | makeresults | eval key="noop" | fields - _time ] | streamstats count as project_no ]
| eval line_merge=case(isnum(project_no), project_no, isnum(result_no), result_no)
| stats first(key) as project, first(total_*) as "total_*" by line_merge | fields - line_merge

| eval pct_total_done=round(total_done/total_issues*100, 2), pct_total_to_do=round(total_to_do/total_issues*100, 2), pct_total_in_progress=round(total_in_progress/total_issues*100, 2)
| foreach pct_* [ eval <<FIELD>> = if(isnull('<<FIELD>>'), "0.00", '<<FIELD>>' ) ]

| where project!="noop"

| eval _time=now() | fields _time, project, pct_*, total_*

is taking a while and not performing properly right?

Yes almost 6 minutes . Also please check below snip this tab is also not populating data. Overview jira service desk

[guilhemmarchand]

Yes almost 6 minutes .

Ok, it is not so bad in the end, if you add at the end of the search a:

collect index="<your summary index>" source="jira_statistics"

Then you save this and schedule to run say every 10 minutes, it would store the result in a summary data, then you can easily build anything you need for reporting and investigation purposes.

It might be possible to improve the search and I need to check for that, I do not know yet.

Question 2:

As I answered earlier several times, are you creating any issue from Splunk to JIRA already? using the alert action.

If not, there will never be any activity here.

Second, you need to have access to the _internal index.

[guilhemmarchand]

Assuming all questions were answered.