search

sipcapture / homer-app

HOMER 7.x Front-End and API Server
http://sipcapture.io
GNU Affero General Public License v3.0
205 stars 84 forks source link

Broken setup when SIPHeader not set #245

Closed kevin-olbrich closed 4 years ago

kevin-olbrich commented 4 years ago

My dashboard does not work correctly. I previously used homer5 and tried to set up latest homer.

The session id has another cursor but I can't click - nothing happens. Also user data isn't shown. Feeders are asterisk v13 and freeswitch 1.8.x.

image

Source and Dest IPs have been removed, they are shown correctly.

lmangani commented 4 years ago

It appears your mapping might be at fault here. I would consider re-populating the homer-app parameters, without dropping the traffic, to see if it helps: homer-app -populate-table-db-config (might need the force bit set, too)

kevin-olbrich commented 4 years ago

I already tried this but it did not help. I just did this again:

root@capt-fe01 ~ # homer-app -populate-table-db-config  -force-populate

HOMER - filling tables for the config DB [dbname=homer_data] 
 reinstalling users 
 reinstalling global_settings 
 reinstalling agent_location_session 
 reinstalling hepsub_mapping_schema 
 reinstalling user_settings 
 reinstalling mapping_schema 
 reinstalling versions 

DONE 
root@capt-fe01 ~ # systemctl restart homer-app

Still same display.

I did another try using a factory new firefox - same issue (no adblock, etc.).

Config:

{
  "auth_settings": {
    "_comment": "The type param can be internal, ldap",
    "type": "internal"
  },
  "database_config": {
    "help": "Settings for PGSQL Database (settings)",
    "host": "localhost",
    "name": "homer_data",
    "node": "LocalNode",
    "pass": "homer_password",
    "user": "homer_user"
  },
  "database_data": {
    "capt01": {
      "help": "Settings for PGSQL Database (data)",
      "host": "192.168.30.3",
      "name": "homer_data",
      "node": "capt01",
      "pass": "secret_password",
      "user": "homer"
    },
    "capt02": {
      "help": "Settings for PGSQL Database (data)",
      "host": "192.168.31.3",
      "name": "homer_data",
      "node": "capt02",
      "pass": "secret_password",
      "user": "homer"
    },
    "localnode": {
      "host": "localhost",
      "name": "homer_config",
      "node": "localnode",
      "pass": "homer_password",
      "user": "homer_user"
    }
  },
  "decoder_shark": {
    "_comment": "Here you can do packet decoding to using tshark application. Please define uid, gid if you run the app under root",
    "active": false,
    "bin": "/usr/bin/tshark",
    "protocols": [
      "1_call",
      "1_registration",
      "1_default"
    ]
  },
  "http_settings": {
    "debug": false,
    "gzip": true,
    "help": "Settings for the HOMER Webapp Server",
    "host": "0.0.0.0",
    "port": 9080,
    "root": "/usr/local/homer/dist"
  },
  "influxdb_config": {
    "database": "homer",
    "help": "Settings for InfluxDB Database (optional)",
    "host": "http://127.0.0.1:8086",
    "name": "homer_config",
    "pass": "influx_password",
    "policy": "autogen",
    "user": "influx_user"
  },
  "ldap_config": {
    "attributes": [
      "givenName",
      "sn",
      "mail",
      "uid"
    ],
    "base": "dc=example,dc=com",
    "binddn": "uid=readonlysuer,ou=People,dc=example,dc=com",
    "bindpassword": "readonlypassword",
    "groupfilter": "(memberUid=%s)",
    "host": "ldap.example.com",
    "port": 389,
    "userfilter": "(uid=%s)",
    "usessl": false
  },
  "system_settings": {
    "_loglevels": "can be: fatal, error, warn, info, debug, trace",
    "help": "Settings for HOMER logs",
    "hostname": "capt-fe01.example.com",
    "loglevel": "error",
    "logname": "homer-app.log",
    "logpath": "/usr/local/homer/log",
    "logstdout": false,
    "uuid": "8c6fc419-22ca-4902-936e-3c0a551b4448"
  }

The "localnode" in "database_data" has been added automatically, it does not realy make sense, isn't it?

kevin-olbrich commented 4 years ago

I've corrected the database connection but the issue is not resolved.

kevin-olbrich commented 4 years ago

Firefox Web Console logs this error:

image Google Chrome, Fedora 31

The file exists and I can click the link inside the message.

Windows does not display anything at all: image Google Chrome, Windows 10 (1909)

lmangani commented 4 years ago

Are you using the latest release 1.1.8? Please update your homer-app binary and provide a step-by-step sequence to reproduce your issue, thanks.

kevin-olbrich commented 4 years ago

Yes, it's the same for 1.1.8:

root@capt-fe01 ~ # apt list homer-app -a
Listing... Done
homer-app/stretch,now 1.1.8 amd64 [installed]
homer-app/stretch 1.1.6 amd64

Host is Debian 10/Buster.

I did nothing special, I set a password for the postgres user using psql and followed these steps: https://github.com/sipcapture/homer-app/blob/master/CONFIGURE.md https://github.com/sipcapture/homer-app/blob/master/README.md

Homer-app then beautified my webapp_config.json and let me login.

kevin-olbrich commented 4 years ago

And this happens when I click on a Session ID: image

adubovikov commented 4 years ago

@kevin-olbrich can you share to us the result of /api/v3/search/call/data ?

adubovikov commented 4 years ago

and do you use heplify-server 1.14 ?

adubovikov commented 4 years ago

looks like this is broken mapping. Can you go to Settings->Mapping->call SIP and paste here the mapping. Also please be sure that you don't have any dupplications! Please use heplify-server 1.14

kevin-olbrich commented 4 years ago

@kevin-olbrich can you share to us the result of /api/v3/search/call/data ?

GET after I am logged in: {"message":"Method Not Allowed"}

Do I need to set some Auth headers via Postman?

and do you use heplify-server 1.14 ?

Yes:

root@capt01 ~ # apt list heplify-server -a
Listing... Done
heplify-server/stretch,now 1.14.0 amd64 [installed]
heplify-server/stretch 1.11-1 amd64
heplify-server/stretch 1.1.3 amd64

looks like this is broken mapping. Can you go to Settings->Mapping->call SIP and paste here the mapping. Also please be sure that you don't have any dupplications! Please use heplify-server 1.14

image

adubovikov commented 4 years ago

If you click on F12 you will able catch all network calls

kevin-olbrich commented 4 years ago

I use real data for the test, I need to make sure I don't leak data.

The data looks like this:

image ... image

kevin-olbrich commented 4 years ago

I'm using HEP from asterisk and freeswitch, no port mirror. Currently the interconnect is using UDP but we want to be able to use TCP/TLS, that's why a port mirror does not make sense here. Does a clean install work for you?

PS: Homer5 was working fine until I shut it down yesterday. I was able to browse sessions and see user names (FROM, etc.), source data should be fine then (IMHO).

lmangani commented 4 years ago

Are you using HEPv3 in your Agent setups? This is extremely important and often overlooked.

kevin-olbrich commented 4 years ago

Are you using HEPv3 in your Agent setups? This is extremely important and often overlooked.

I think so:

root@sip01:/etc/freeswitch# grep -r "hep"
autoload_configs/sofia.conf.xml:    <param name="capture-server" value="udp:192.168.30.3:9060;hep=3;capture_id=1001"/>
kevin-olbrich commented 4 years ago

Ok, we are getting closer. As I did not fully understand what SIPHeader in heplify-server does, I left it on default: []. Using the demo setting, I now see data and JS started working (maybe the code depends on it?).

Why do I need to set it? For debug, I would like to see "everything". Is this different to h5?

adubovikov commented 4 years ago

so, once you will define the SIPHeader, it will extract only these headers. Better keep it empty!

also, please update your mapping:

./homer-app -populate-table-db-config -populate-table=mapping_schema -force-populate

lmangani commented 4 years ago

Shall we consider this resolved? Sounds to me the issue was simply the misconfigured SIPHeader in heplify-server

kevin-olbrich commented 4 years ago

Why do I need to set SIPHeader? I did not set it ( [] ) and it did not work. From your comment it sounds like, an empty value should not filter the data but it looks like, it just drops everything if it's unset.

Better keep it empty!

That's the problem :-) I've not tried what happens when I set it back to empty but on my fresh install, that field was empty and it caused this issue.

lmangani commented 4 years ago

@negbie do you mind shedding some light here? This was implemented w/o much documentation indeed

kevin-olbrich commented 4 years ago

Just to make sure: It started to work when I added Headers to that setting, not the other way around.

adubovikov commented 4 years ago

can you paste your toml ? I just checked on our lab and it works without that param :-/

negbie commented 4 years ago

I think it's pretty obvious what his problem is. He set the value intentionally to an empty array. I will prevent this later by checking if len of SIPHeaders is >1-5 and if not use defaults.

kevin-olbrich commented 4 years ago

I think it's pretty obvious what his problem is. He set the value intentionally to an empty array. I will prevent this later by checking if len of SIPHeaders is >1-5 and if not use defaults.

Exactly! Just like the example config.

HEPAddr               = "0.0.0.0:9060"
HEPTCPAddr            = "0.0.0.0:9060"
HEPTLSAddr            = "0.0.0.0:9061"
ESAddr                = ""
ESDiscovery           = false
LokiURL               = ""
LokiBulk              = 200
LokiTimer             = 4
LokiBuffer            = 100000
LokiHEPFilter         = []
ForceHEPPayload       = []
PromAddr              = ""
PromTargetIP          = ""
PromTargetName        = ""
DBShema               = "homer7"
DBDriver              = "postgres"
DBAddr                = "localhost:5432"
DBUser                = "homer"
DBPass                = "xxxxxxxxxxxxx"
DBDataTable           = "homer_data"
DBConfTable           = "homer_config"
DBBulk                = 200
DBTimer               = 4
DBBuffer              = 400000
DBWorker              = 8
DBRotate              = true
DBPartLog             = "2h"
DBPartSip             = "1h"
DBPartQos             = "6h"
DBDropDays            = 14
DBDropDaysCall        = 0
DBDropDaysRegister    = 0
DBDropDaysDefault     = 0
DBDropOnStart         = false
Dedup                 = true
DiscardMethod         = []
AlegIDs               = ["X-CID","P-Charging-Vector,icid-value=\"?(.*?)(?:\"|;|$)","X-BroadWorks-Correlation-Info"]
CustomHeader          = []
SIPHeader            = []
LogDbg                = ""
LogLvl                = "info"
LogStd                = true
LogSys                = true
Config                = "./heplify-server.toml"
ConfigHTTPAddr        = ""

# Examples:
# -------------------------------------
# ESAddr          = "http://127.0.0.1:9200"
# DBShema         = "homer7"
# DBDriver        = "postgres"
# LokiURL         = "http://localhost:3100/api/prom/push"
# LokiHEPFilter   = [1,5,100]
# PromAddr        = "0.0.0.0:8899"
# PromTargetIP    = "10.1.2.111,10.1.2.4,10.1.2.5,10.1.2.6,10.12.44.222"
# PromTargetName  = "sbc_access,sbc_core,kamailio,asterisk,pstn_gateway"
# AlegIDs         = ["X-CID","P-Charging-Vector,icid-value=\"?(.*?)(?:\"|;|$)","X-BroadWorks-Correlation-Info"]
# DiscardMethod   = ["OPTIONS","NOTIFY"]
# CustomHeader    = ["X-CustomerIP","X-Billing"]
# SIPHeader      = ["callid","method","ruri_user","ruri_domain","from_user","from_domain","from_tag","to_user","to_domain","to_tag","via","contact_user"]
# LogDbg          = "hep,sql"
# LogLvl          = "warning"
adubovikov commented 4 years ago

and if you will remove SIPHeader and restart heplify-server ? Will it work ?

On Thu, 23 Jan 2020 at 13:11, Kevin Olbrich notifications@github.com wrote:

I think it's pretty obvious what his problem is. He set the value intentionally to an empty array. I will prevent this later by checking if len of SIPHeaders is >1-5 and if not use defaults.

Exactly! Just like the example config.

HEPAddr = "0.0.0.0:9060" HEPTCPAddr = "0.0.0.0:9060" HEPTLSAddr = "0.0.0.0:9061" ESAddr = "" ESDiscovery = false LokiURL = "" LokiBulk = 200 LokiTimer = 4 LokiBuffer = 100000 LokiHEPFilter = [] ForceHEPPayload = [] PromAddr = "" PromTargetIP = "" PromTargetName = "" DBShema = "homer7" DBDriver = "postgres" DBAddr = "localhost:5432" DBUser = "homer" DBPass = "xxxxxxxxxxxxx" DBDataTable = "homer_data" DBConfTable = "homer_config" DBBulk = 200 DBTimer = 4 DBBuffer = 400000 DBWorker = 8 DBRotate = true DBPartLog = "2h" DBPartSip = "1h" DBPartQos = "6h" DBDropDays = 14 DBDropDaysCall = 0 DBDropDaysRegister = 0 DBDropDaysDefault = 0 DBDropOnStart = false Dedup = true DiscardMethod = [] AlegIDs = ["X-CID","P-Charging-Vector,icid-value=\"?(.*?)(?:\"|;|$)","X-BroadWorks-Correlation-Info"] CustomHeader = [] SIPHeader = [] LogDbg = "" LogLvl = "info" LogStd = true LogSys = true Config = "./heplify-server.toml" ConfigHTTPAddr = ""

Examples:

-------------------------------------

ESAddr = "http://127.0.0.1:9200"

DBShema = "homer7"

DBDriver = "postgres"

LokiURL = "http://localhost:3100/api/prom/push"

LokiHEPFilter = [1,5,100]

PromAddr = "0.0.0.0:8899"

PromTargetIP = "10.1.2.111,10.1.2.4,10.1.2.5,10.1.2.6,10.12.44.222"

PromTargetName = "sbc_access,sbc_core,kamailio,asterisk,pstn_gateway"

AlegIDs = ["X-CID","P-Charging-Vector,icid-value=\"?(.*?)(?:\"|;|$)","X-BroadWorks-Correlation-Info"]

DiscardMethod = ["OPTIONS","NOTIFY"]

CustomHeader = ["X-CustomerIP","X-Billing"]

SIPHeader = ["callid","method","ruri_user","ruri_domain","from_user","from_domain","from_tag","to_user","to_domain","to_tag","via","contact_user"]

LogDbg = "hep,sql"

LogLvl = "warning"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sipcapture/homer-app/issues/245?email_source=notifications&email_token=ABCN2JP2XA6ZPFU5FVUS3VDQ7GCQPA5CNFSM4KJ5LGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJXFC6A#issuecomment-577655160, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCN2JIHD2Z3RPKY4U5CXTDQ7GCQPANCNFSM4KJ5LGPQ .

negbie commented 4 years ago

@kevin-olbrich this should be fixed now. Just to make clear what SIPHeader does. In homer7 you have a column data_header. The SIPHeader in heplify-server.toml let's you pick all the SIPHeader you want. Currently following are implemented in heplify-server:

callid,method,ruri_user,ruri_domain,from_user,from_domain,from_tag,to_user,to_domain,to_tag,via,contact_user,contact_domain,user_agent,pid_user,auth_user,server,content_type,reason,diversion,expires

I don't know how far the homer7 GUI is but at some point you should be able to pick the header you want to be able to search. For obvious reasons callid is quite important.

negbie commented 4 years ago

If you set SIPHeader = [] or don't set it at all, I will use following default header: ruri_user,ruri_domain,from_user,from_domain,to_user,callid,method,user_agent

adubovikov commented 4 years ago

https://github.com/sipcapture/heplify-server/releases/tag/1.15 https://github.com/sipcapture/homer-app/releases/tag/1.1.12

adubovikov commented 4 years ago

https://github.com/sipcapture/homer-app/releases/tag/1.1.15

adubovikov commented 4 years ago

I will close the ticket. Please reopen if needed

kevin-olbrich commented 4 years ago

I'll try that next week. Thanks for taking a look into this issue!