search

sipcapture / homer

HOMER - 100% Open-Source SIP, VoIP, RTC Packet Capture & Monitoring
https://sipcapture.org
GNU Affero General Public License v3.0
1.65k stars 243 forks source link

Fresh install, INFLUXDB CHART shows No Data #369

Closed SB-JohnK closed 4 years ago

SB-JohnK commented 5 years ago

I used homer installer, I installed influxdb and grafana.

I configured a scraper in influxdb to point to heplify-server http://192.168.8.18:9096/metrics

I can use influxdb data explorer to see rtcp counters so I believe data collection is working.

I configured my timezone in the world clock.

I get 'No Data' from INFLUXDB CHART' on either the 'Home' or 'RTC' dashboards.

What should these graphs be showing? Do I need to configure them?

I would like graphs showing jitter and packet loss at each of my capture nodes.

If I look at the system journal I see the errors logged for homer-app

Oct 15 09:43:40 homer.simplybits.net homer-app[4377]: Error.... Oct 15 09:43:40 homer.simplybits.net homer-app[4377]: Debug: handler, error Oct 15 09:43:40 homer.simplybits.net homer-app[4377]: {"msec":3.150979000143707,"error":"stats data has not been found","data":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"stats data has not been found"},"headers":{}}}}

lmangani commented 5 years ago

Hi,

Sounds like a connectivity issue between the UI and the backend potentially. Please try reconfiguring the influxdb widget, and see if you get any dropdown values when selecting the Databases / Series as a first step - please update the ticket with results.

SB-JohnK commented 5 years ago

I do see drop downs but only one option in each.

I've checked and selinux is disabled and firwalld is not running, I have tcp ports 80, 9999 and 9096 permitted through my hardware firewall.

homer1 homer2

aqsyonas commented 5 years ago

@SB-JohnK please, make sure you have correct details at https://github.com/sipcapture/homer-app/blob/master/server/config.js#L33 I will check the installer and fix it. If not getting properly set.

SB-JohnK commented 5 years ago

I modified config.js to change the port number to 9999, I also found a reference to influx in server.js and changed the port there, still seeing the same results.

SB-JohnK commented 5 years ago

Is there additional troubleshooting I can do at this point to help? provide configurations or logs?

lmangani commented 5 years ago

Hi @SB-JohnK first let's check if there's any data at all. You should have Chronograf installed and running on port 8888 on the same host. Try access it and see if there's any database from homer, and it it contains any of your data. If this works, its a connectivity issue betwen the (now old) UI and InfluxDB, otherwise it's an issue somewhere in the heplify-server -> telegraf -> influxdb path

TLDR; unless the homer database exists and has fresh data, its a connectivity issue UI -> InfluxDB

SB-JohnK commented 5 years ago

I'm unable to access anything on port 8888.

Netstat doesn't show anything listening on 8888 either:


[root@homer ~]# netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name                                                     
tcp        0      0 127.0.0.1:4222          0.0.0.0:*               LISTEN      4463/influxd                                                         
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      25733/node                                                           
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      5213/sshd                                                            
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      3973/postmaster                                                      
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      5725/master                                                          
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      25733/node                                                           
tcp6       0      0 :::9060                 :::*                    LISTEN      4562/heplify-ser                                                     ver
tcp6       0      0 :::9096                 :::*                    LISTEN      4562/heplify-ser                                                     ver
tcp6       0      0 :::9999                 :::*                    LISTEN      4463/influxd                                                         
tcp6       0      0 :::22                   :::*                    LISTEN      5213/sshd                                                            
tcp6       0      0 ::1:5432                :::*                    LISTEN      3973/postmaster                                                      
tcp6       0      0 ::1:25                  :::*                    LISTEN      5725/master

I am able to access the influxdb web interface on port 9999, I can use the data explorer and see the following measurements that appear to be homer related:

heplify_packets_zie heplify_packets_total heplify_rtcp_dlsr heplify_rtcp_fraction_lost heplify_rtcp_jitter heplify_rtcp_packets_lost

I've attached a screenshot influxdb png

lmangani commented 5 years ago

Oh - you're using the TICK stack 2.0. I guess it's as simple as changing the Influxdb settings of the HOMER-APP which currently points at 8086 rather than 9999 as I guess it should be. Could you give this a shot?

https://github.com/sipcapture/homer-app/blob/master/server/config.js#L32

SB-JohnK commented 5 years ago

I changed config.js and server.js both when @aqsyonas mentioned it last week, please see attached:

I had to rename the files to .txt to upload them. server.txt config.txt

SB-JohnK commented 5 years ago

@lmangani Any more suggestions? Changing the settings and restarting did not make a difference.

rossworth commented 5 years ago

I have the exact same issue. Have also made the port changes in the files. I also used the installer script.

rossworth commented 5 years ago

I have tried the installer script on Debian 9 and centos 7, I have updated the ports in the two files mentioned above. Both operating systems have the same issue @SB-JohnK did you get any further? @lmangani do you have any other suggestions to try? I do see data in InfluxDB. Any help greatly appreciated.

lmangani commented 5 years ago

I am travelling and can't test this but I guess we just need to provide the correct endpoint for InfluxQL HTTP queries to the API - play with something like the following

const influxdb = {
  host: '127.0.0.1/api/v2/query',
  port: 9999,
  database: 'homer'
}

FYI @rossual and all, we're about to release 7.7 and we'll nail this down permanently

rossworth commented 5 years ago

Thanks @lmangani I'm also travelling now, I will test on thursday next week. Thanks again for the response. Looking forward to 7.7

rossworth commented 5 years ago

@lmangani I tried the suggestion above and /api/v2 but no luck. How long until the 7.7 release? Is there anything I can provide/test?

lmangani commented 4 years ago

Gents, 7.7 is already out in alpha and i invite you all to switch to it and help us bootstrap it before the end of the month.

We can easily troubleshoot and fix issues (shall any be left) and since the UI is completely changed, and the access method to statistics did too (we now also allow embedding Grafana widgets from dashboards alongside raw from influx and prometheus)

sebky94 commented 4 years ago

Hello,

Thank you @lmangani for the update. I'll try to reinstall my server.

Do you guys think it's normal that the service on which the ports the server listens on are 9060 and 9096 is tcp6:

tcp6 0 0 :::9060 ::: LISTEN 4562/heplify-ser ver tcp6 0 0 :::9096 ::: LISTEN 4562/heplify-ser ver

i have the same on my server.

lmangani commented 4 years ago

This is a common misunderstanding, I'm quite certain tcp6 sockets can also listen to tcp4 when operating on all interfaces as per this example. If you can receive packets via ipv4 this is implicit.

sebky94 commented 4 years ago

Thank you @lmangani .

Is 7.7 available for CentOS7 too?

lmangani commented 4 years ago

The 7.7 installer should support both CentOS and Debian just fine. I would although suggest running one of the 7.7 Docker bundles to get the full stack results on a single server and still be able to move things around at a later time as those (particularly the Prometheus bundle) come pre-provisioned for Dashboards, etc.

sebky94 commented 4 years ago

Is there a dedicated guide to install the 7.7?

Is uploading and executing the install enough ? or should i go through a series of installs seperatly (for example, the DB, HEPlify ...)?

lmangani commented 4 years ago

I'm not sure what the confusion is, but there's a homer-installer branch 7.7 with the current changes - this should spin up a 7.7 instance you can update/upgrade over time. The best method for those unfamiliar with the process is mimic what our Docker set do. Let me know if you have a more precise question ;)

SB-JohnK commented 4 years ago

I had some trouble with the 7.7 homer installer branch, It may be something weird to my environment but there were several places in the script where Yum should have been prompting 'Is this ok [y/d/n]' or 'Is this ok [y/N]' where the question did not display until I had blindly entered 'Y'.

Homer-app is still logging that it can't get stats data:

Dec 05 12:36:12 homer homer-app[17757]: {"msec":5.2992939949035645,"error":"stats data has not been found","data":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"stats data has not been found"},"headers":{}}}} Dec 05 12:38:29 homer homer-app[17757]: Debug: handler, error Dec 05 12:38:29 homer homer-app[17757]: {"msec":0.35391199588775635,"error":"Not Found","data":{"data":null,"isBoom":true,"isServer":false,"output":{"statusCode":404,"payload":{"statusCode":404,"error":"Not Found","message":"Not Found"},"headers":{}},"message":"Not Found"}} Dec 05 12:38:30 homer influxd[23692]: ts=2019-12-05T19:38:30.001098Z lvl=info msg="failed to store statistics" log_id=0JXrWzr0000 service=monitor error=timeout Dec 05 12:38:31 homer homer-app[17757]: AAL { '127.0.0.1:5060': 'localhost' } Dec 05 12:38:40 homer influxd[23692]: ts=2019-12-05T19:38:40.001694Z lvl=info msg="failed to store statistics" log_id=0JXrWzr0000 service=monitor error=timeout Dec 05 12:38:41 homer homer-app[17757]: AAL { '127.0.0.1:5060': 'localhost' }

I'm able to access the chronograph UI but I'm not able to configure the scapper, according to the install output I should be able to add http://192.168.8.18:9096 but I get an error contacting source when I try that.

Trying to access just http://192.168.8.18:9096 gives me a 404 response.

Niii commented 4 years ago

Fresh install of Homer7.7 on Centos7.7, still no data. I can push sip and rtcp and show them with a research, but I can't configure a chart.

When I try to configure an Influxdb chart, got error "db was not found".

sebky94 commented 4 years ago

Same thing here, except that i have no data at all, I have fresh install homer 7.7 nd still no data.

lmangani commented 4 years ago

No data at all, means you also checked directly in InfluxDB if there is data, or just referring to the Homer UI?

sebky94 commented 4 years ago

I meant only the UI, how can i check directly in InfluxDB?

lmangani commented 4 years ago

If you're running InfluxDB 2.0 here's a good pointer: https://v2.docs.influxdata.com/v2.0/get-started/

You might want to configure a scraper for heplify-server:9060/metrics

Niii commented 4 years ago

@sebky94 do you have some traffic coming on port 5060 ? On Centos, network card wasn't on promisc mode, so you receive nothing and heplify client can't grab data.

sebky94 commented 4 years ago

@lmangani Thank you i will do it.

@Niii On the heplify client i have SIP data on port 5060, it's an on production SBC.

lmangani commented 4 years ago

Please upgrade to the latest homer-app and version 7.7 to retest - your issue should be resolved!