search

containerpope / nifi-prometheus-reporter

A reporting task in Nifi which is capable of sending monitoring statistics as prometheus metrics to a prometheus pushgateway.
Apache License 2.0
52 stars 32 forks source link

Using Reporter on Openshift Cluster #4

Closed KugelschreiberY closed 6 years ago

KugelschreiberY commented 6 years ago

Hi there,

thanks for implementing this reporter! Currently I am trying to run it on an openshift cluster (which I am quite new to, hence sorry if this is the wrong place to ask). Unfortunately the reporter does not send the metrics to pushgateway, although I added the built nifi-prometheus-nar-1.6.0.nar file to the lib directory in the Dockerfile by

COPY ./nifi-conf/nifi-prometheus-nar-1.6.0.nar /opt/nifi/lib/nifi-prometheus-nar-1.6.0.nar

Prometheus, pushgateway and nifi are running in three pods respectively; the connection between pushgateway and prometheus seems to be fine. Therefore I expect the problem lies in the set up of the nifi pod. Since I copy the nar-file to nifi beforehand, I do not restart nifi as you mention in your README.

Did I miss something crucial? Again, sorry if this is more of an openshift problem. Thanks in advance.

containerpope commented 6 years ago

The processor itself should send to a url that points to a pushgateway. If you use Openshift you have to create a pushgateway deployment and a service. The name of the service can than be used as target. Hope this helps. If I find the time, I provide an example.

Regards Matthias

Am 07.05.2018 um 08:11 schrieb Simon Gabriel notifications@github.com:

Hi there,

thanks for implementing this reporter! Currently I am trying to run it on an openshift cluster (which I am quite new to, hence sorry if this is the wrong place to ask). Unfortunately the reporter does not send the metrics to pushgateway, although I added the built nifi-prometheus-nar-1.6.0.nar file to the lib directory in the Dockerfile by

COPY ./nifi-conf/nifi-prometheus-nar-1.6.0.nar /opt/nifi/lib/nifi-prometheus-nar-1.6.0.nar

Prometheus, pushgateway and nifi are running in three pods respectively; the connection between pushgateway and prometheus seems to be fine. Therefore I expect the problem lies in the set up of the nifi pod. Since I copy the nar-file to nifi beforehand, I do not restart nifi as you mention in your README.

Did I miss something crucial? Again, sorry if this is more of an openshift problem.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

KugelschreiberY commented 6 years ago

Where/How do I set the pushgateway service as target of my nifi configuration? An example would be great!

containerpope commented 6 years ago

First it depends if your pushgateway is running inside or outside of the openshift cluster. If it should run internally you should be able to deploy the pushgateway via the deploy image option in openshift web ui. After you deployed it, Openshift should have created at least a service and a deploymentconfig. The service can be seen as an cluster internal loadbalancer. If you create for example 3 Pods of your pushgateway, they are all reachable under the services url inside the cluster.

This adress is then reachable from other pods, so for example if your pushgateway service is called pushgateway-nifi and the port is 9000, you can configure the processor in nifi with the dns name http://pushgateway-nifi:9000

Useful links: Kubernetes Service

KugelschreiberY commented 6 years ago

Pushgateway is running internally. Correct me if I'm wrong: the ReportingTask is a processor I have to include in Nifi, there I can set the properties to push the metrics to pushgateway? Then my openshift cluster is fine, I think. However, I cannot find the processor in the webUI list 'Add Processor' although the .nar file is included in the lib directory. Any idea? Thanks!

containerpope commented 6 years ago

Ah ok. You have to use the upper right settings. Nifi Reporting tasks are global and not deployed as processors. See the following guide: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_user-guide/content/Reporting_Tasks.html https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_user-guide/content/Reporting_Tasks.html Hope it helps :)

Am 14.05.2018 um 10:54 schrieb Simon Gabriel notifications@github.com:

Pushgateway is running internally. Correct me if I'm wrong: the ReportingTask is a processor I have to include in Nifi, there I can set the properties to push the metrics to pushgateway? Then my openshift cluster is fine, I think. However, I cannot find the processor in the webUI list 'Add Processor' although the .nar file is included in the lib directory. Any idea? Thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mkjoerg/nifi-prometheus-reporter/issues/4#issuecomment-388744602, or mute the thread https://github.com/notifications/unsubscribe-auth/ALhVGRAvnq0jf3Wxsvtu7pbOfqeAMwntks5tyUYygaJpZM4T0cuN.

KugelschreiberY commented 6 years ago

Thanks a lot. It is working now!

containerpope commented 6 years ago

Always happy to help :)

Am 17.05.2018 um 09:50 schrieb Simon Gabriel notifications@github.com:

Thanks a lot. It is working now!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mkjoerg/nifi-prometheus-reporter/issues/4#issuecomment-389777193, or mute the thread https://github.com/notifications/unsubscribe-auth/ALhVGT5XgS0kND7wtmpSOVyVAxO4klKXks5tzSvFgaJpZM4T0cuN.

NarvekarSwati commented 5 years ago

how u created nifi-prometheus-nar-1.6.0.nar file