Status History

[idglabs]

Hello

I'm wondering if is possible to have in spunk dashboard the Status History of some processors? using all information from NIFI API

[avasquezkudaw]

Hi, sorry for being offline for a while. It might be possible as long as the information is obtained from the API. Do you know which endpoint could be used to obtain it?

[idglabs]

Welcome back.

Thank you for your reply.

• Yes that's the idea to use the information obtained from the API.
  • to have a locally installed a python script on every nifi instance in the cluster which can produce the JSON file from API and save in .nifi/logs/ folder
• a new dashboard where from pop down menu to choose the processors and view the status history of that.
• ENDPOINT I think should be in the csv file or configures in a nifi processors (in order to be customize and changed as needed)

[avasquezkudaw]

Hi,

I found this endpoint in the nifi api (/nifi-api/flow/process-groups/root/status?recursive=true). Can you test if running it in your environment returns results? On my side it delivered results in json format.

The problem with this is the complexity of the query in splunk since it must search all processors recursively at multiple levels. I'm thinking of ideas to visually display these states.

[idglabs]

Hi ,

Thank you for your feedback.

I've tested your ENDPOINT -(/nifi-api/flow/process-groups/root/status?recursive=true

it's return huge data: Results in json format.

You are very right related to complexity of query splunk and for this I'm thinking to use the API ENDPOINT only for selected processor Group ,(which can be defined in CSV file example hows defined instances in splunk drop-down menu)

I've already start to work on some possible scenario (but still on paper work )

so if my thought it's right in order to have the API ENDPOINT for Status History of some Processor there are the steps

• API to be interrogate should be /flow/process-groups/{id}/status/history , where /{id} it's the same ID which show on Status History when right of the processor name and select History
• splunk drop-down menu with all processor list
• splunk drop-down menu generated automatically base of the previous selection (processor)

what do you think?

[avasquezkudaw]

I think it's a good idea. I checked that endpoint and it replies a lot of data too. My question is how often the endpoint should be consulted, and how the end user can set the processid list. I am going to check if setting in a csv or setting in the modular input may be the best alternative. If you have another idea, welcome. Regarding the information, I think this should be displayed as a line graph over time. You tell me your opinions. Greetings.

[idglabs]

Sorry for the late reply. I'm working on some script .python for API interrogation.

• ENDPOINT will be interrogated every 10/15 minutes (still investigating and adjusting where necessary)

• processor ID, one possibility it's to generate root and output will be converted in Lookup csv (or will be generated only on-demand first time or when the workflow it's has some changed.) this will be used in Splunk Dashboard. mapping_lookup.json

• metrics.json is API results after interrogate API base of the defined Parent ID (in order to not query all flows)

• yes your suggestion for displayed as a line graph over time is very good idea.

• my idea is to show something similar like Status History graphics and numbers , like per image

• 

Thank you very much for your support.

One I have something I will let you know

[avasquezkudaw]

Hello,

Maybe it's a good idea to add a new endpoint in nifi_TA_monitoring script. I'm working in datainput page, and I added selectors for endpoints and list of process groups to monitor:

Tell me what do you think about it.

For other side, the event of this endpoint must be splitted in splunk, because one event have multiples timestams and values. So, i don't know if you have examples of splunk query to create a metric using this information.

Regards.

[avasquezkudaw]

In new version, we added status history monitoring.