how to push additionnal metrics

[julienlau]

Hi, I appreciate your work, but I was more willing to expand the capability of the "status history" per processor to be able to see usage during a larger timerange than 5 minutes. Is there any chance that this reporter could expose some metrics from nifi.components.status.repository ?

If not, could you confirm that in order to add new metrics, you just use vanilla pushgateway and use a HTTP post processor within nifi ? If so, any chance to have a template of such POST ?

Thanks and regards, Julien

[rihtak06]

Did you get metrics? i have put nar file inside lib but i didnt get any metrics in pushgateway? can you give me steps how you got it?

[julienlau]

I don't have any nifi specific metrics. Just system info about go and process.

[rihtak06]

Any metrics you got a process group?

[containerpope]

Hi, I appreciate your work, but I was more willing to expand the capability of the "status history" per processor to be able to see usage during a larger timerange than 5 minutes. Is there any chance that this reporter could expose some metrics from nifi.components.status.repository ?

If not, could you confirm that in order to add new metrics, you just use vanilla pushgateway and use a HTTP post processor within nifi ? If so, any chance to have a template of such POST ?

Thanks and regards, Julien

A nifi reporting task is more of a global metrics generator. In our case, this reporter will provide metrics of the underlying jvm and the metrics you can also see in the UI for a process group. If you don't select a process group, you get the global metrics from all together. If you don't want these generic metrics from nifi, but rather some custom ones, you have to do exactly what you proposed. An example would be to use a PostHttp processor or something similar and transmit the metrics directly in the flow.

At the moment the following metrics are available:

public interface MetricNames {

    // Metric Name separator
    String METRIC_NAME_SEPARATOR = ".";

    // NiFi Metrics
    String FLOW_FILES_RECEIVED = "FlowFilesReceivedLast5Minutes";
    String BYTES_RECEIVED = "BytesReceivedLast5Minutes";
    String FLOW_FILES_SENT = "FlowFilesSentLast5Minutes";
    String BYTES_SENT = "BytesSentLast5Minutes";
    String FLOW_FILES_QUEUED = "FlowFilesQueued";
    String BYTES_QUEUED = "BytesQueued";
    String BYTES_READ = "BytesReadLast5Minutes";
    String BYTES_WRITTEN = "BytesWrittenLast5Minutes";
    String ACTIVE_THREADS = "ActiveThreads";
    String TOTAL_TASK_DURATION_SECONDS = "TotalTaskDurationSeconds";
    String TOTAL_TASK_DURATION_NANOS = "TotalTaskDurationNanoSeconds";

    // JVM Metrics
    String JVM_UPTIME = "jvm.uptime";
    String JVM_HEAP_USED = "jvm.heap_used";
    String JVM_HEAP_USAGE = "jvm.heap_usage";
    String JVM_NON_HEAP_USAGE = "jvm.non_heap_usage";
    String JVM_THREAD_STATES_RUNNABLE = "jvm.thread_states.runnable";
    String JVM_THREAD_STATES_BLOCKED = "jvm.thread_states.blocked";
    String JVM_THREAD_STATES_TIMED_WAITING = "jvm.thread_states.timed_waiting";
    String JVM_THREAD_STATES_TERMINATED = "jvm.thread_states.terminated";
    String JVM_THREAD_COUNT = "jvm.thread_count";
    String JVM_DAEMON_THREAD_COUNT = "jvm.daemon_thread_count";
    String JVM_FILE_DESCRIPTOR_USAGE = "jvm.file_descriptor_usage";
    String JVM_GC_RUNS = "jvm.gc.runs";
    String JVM_GC_TIME = "jvm.gc.time";
}

If the metrics you want are part of a process group, you can extract them from ProcessGroupStatus. If you submit a PR we can implement them into the project.

[julienlau]

Ok then, there is a problem with the installation process, because I did not have these metrics. NB: I did not use docker. and used nifi 1.8.0 The only metrics I have are go_gc_duration_seconds{quantile="0"} go_gc_duration_seconds{quantile="0.25"} go_gc_duration_seconds{quantile="0.5"} go_gc_duration_seconds{quantile="0.75"} go_gc_duration_seconds{quantile="1"} go_gc_duration_seconds_sum go_gc_duration_seconds_count go_goroutines go_info{version="go1.11.1"} go_memstats_alloc_bytes go_memstats_alloc_bytes_total go_memstats_buck_hash_sys_bytes go_memstats_frees_total go_memstats_gc_cpu_fraction go_memstats_gc_sys_bytes go_memstats_heap_alloc_bytes go_memstats_heap_idle_bytes go_memstats_heap_inuse_bytes go_memstats_heap_objects go_memstats_heap_released_bytes go_memstats_heap_sys_bytes go_memstats_last_gc_time_seconds go_memstats_lookups_total go_memstats_mallocs_total go_memstats_mcache_inuse_bytes go_memstats_mcache_sys_bytes go_memstats_mspan_inuse_bytes go_memstats_mspan_sys_bytes go_memstats_next_gc_bytes go_memstats_other_sys_bytes go_memstats_stack_inuse_bytes go_memstats_stack_sys_bytes go_memstats_sys_bytes go_threads process_cpu_seconds_total process_max_fds process_open_fds process_resident_memory_bytes process_start_time_seconds process_virtual_memory_bytes process_virtual_memory_max_bytes promhttp_metric_handler_requests_in_flight promhttp_metric_handler_requests_total{code="200"} promhttp_metric_handler_requests_total{code="500"} promhttp_metric_handler_requests_total{code="503"} pushgateway_build_info{branch="HEAD",goversion="go1.11.1",revision="290ff867b4e55aad161e56786740802779d4de11",version="0.6.0"}

[containerpope]

Sorry for the late response, had some other projects going on recently. I updated the docs, you should see the metrics after enabling the Reporting Task directly inside the Pushgateway. .

Maybe the connection between your Pushgateway and Nifi isn't working or something is misconfigured?

...
# TYPE jvm_general_stats gauge
jvm_general_stats{instance="6acfd1a0ca0d",job="jvm_global",status="file_descriptor"} 0.0025815963745117188
jvm_general_stats{instance="6acfd1a0ca0d",job="jvm_global",status="total_committed"} 7.07657728e+08
jvm_general_stats{instance="6acfd1a0ca0d",job="jvm_global",status="total_init"} 5.39426816e+08
jvm_general_stats{instance="6acfd1a0ca0d",job="jvm_global",status="total_max"} 5.36870911e+08
jvm_general_stats{instance="6acfd1a0ca0d",job="jvm_global",status="total_used"} 3.78900144e+08
jvm_general_stats{instance="6acfd1a0ca0d",job="jvm_global",status="uptime"} 1804
...

Closing this for now, because I can't find any issues with the reporting task itself.

[usamaB]

@mkjoerg Hi, In my use case, I want to monitor failure queues. Let's say if the count reaches a threshold, I want to send the info(count, flow file info(can be errors)) to Prometheus. I have the flow for it but don't know how to expose a /metrics endpoint in NiFi and convert the data into Prometheus format in NiFi.

Second, if what should be a sample POST processer to send data to Gateway. The simple one gives 405 Method not allowed.

Any lead would be greatly appreciated. I'm relatively new to NiFi.

Thanks