How To search and visualize LOGs from PGSQL (missing HEP Logs subtab)?

[gmaruzz]

Hello,

I installed latest homer7 from deb packages

I would like not to use Loki at all, completely rely on postgrsql

I have log entries in postgresql, and I can search and find in smartsearch, visualization is correct

How I can have the Logs tab in the result widget of SIP search to display the entries from PostgreSQL instead of trying to query a non existent Loki?

(also, I only have SIP and RTCP choices when I try to allow log entries in messages list, probably is related to the Loki hijacking of logs :) )

[lmangani]

Hi Giovanni,

In Homer 7 the feature is completely optional and does nothing unless enabled. You are not forced to use it at all, and even if you do all data still gets stored in Postgres when configured. For the logs tab to work you'd simply have to send HEP Logs. There's no hijacking involved.

Are you positive you're actually sending HEP logs type 100 from OpenSIPS?

[gmaruzz]

Hello Lorenzo,

thanks a lot for the prompt answer!!

Unfortunately in my case seems to not be possible :(

I installed H7 from latest deb packages

I have logs in DB, and I can even search and see them, if I use smartsearch dash

In tab Logs, that you screenshotted, I only have Loki "subtab", I do not have HEP Logs "subtab"

So, is probably a matter of configuration...

Would you hint me which config would make the HEP Logs "subtab" to disappear?

I even tried to disable Loki in both the Heplify-server and the homer-app config, but nothing has changed: I only have the Loki "subtab", no HEP logs "subtab"...

[gmaruzz]

(Yes, I'm positive I sent as type 100, and they are in the correct db table)

homer_data=# select * from hep_proto_100_default;
 id |           sid            |          create_date          |                                                                                                                             protocol_header                                                                                                                             |             data_header              |          raw          
----+--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+-----------------------
  9 | 3_926387462@192.168.1.22 | 2023-09-23 08:17:42.005139+00 | {"dstIp": "192.168.1.61", "srcIp": "192.168.1.61", "dstPort": 35349, "srcPort": 35349, "protocol": 17, "captureId": "13041664", "payloadType": 100, "timeSeconds": 1695457062, "timeUseconds": 5139, "correlation_id": "3_926387462@192.168.1.22", "protocolFamily": 2} | {"node": "13041664", "proto": "log"} | ciao ciao
(2 rows)

[gmaruzz]

(I sent them as test from hepipe)

[gmaruzz]

(I now reinstalled from deb packages, using the install script. I first apt-get purged the packages and deleted the configs, and dropped both databases from pgsql'. They were recreated and populated during install. Problem seems to still exists: no HEP Logs "subtab", only Loki "subtab")

[gmaruzz]

(I am using https://github.com/sipcapture/homer-installer on bullseye)

[lmangani]

Hey @gmaruzz if you have HEP 100 logs in the database and you can't see them, most likely its because there's no correlation_id set pointing at the SIP session you're trying to read them from. If the correlation_id is there and matches, then mapping config has broken.

[gmaruzz]

I see I got this error in js console (address redacted):

polyfills.39594fe71d4c206db379.js:1 POST https://123.4.5.6:5030/api/v3/agent/search/1dc62be9-bfab-4d71-a18e-1480205821c3/cdr 400 (Bad Request)

[gmaruzz]

also, if I use the "smartsearch" dashboad, I can search in logs, and they are visualized correctly

[lmangani]

Ciao Gio, I'm sure you can search for them, but without a correlation_id they cannot be automatically displayed as part of sessions. Could you confirm if your logs (hep 100) have a correlation header matching your SIP Call? Feel free to add an example.

[gmaruzz]

homer_data=# select * from hep_proto_100_default where sid='3_2195560193@192.168.1.22' limit 1;
 id |            sid            |          create_date          |                                                                                                                             protocol_header                                  
                                                                                            |             data_header              |        raw        
----+---------------------------+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------+--------------------------------------+-------------------
  2 | 3_2195560193@192.168.1.22 | 2023-09-23 16:49:03.006467+00 | {"dstIp": "192.168.1.61", "srcIp": "192.168.1.61", "dstPort": 35349, "srcPort": 35349, "protocol": 17, "captureId": "13041664", "payloadType": 100, "timeSeconds": 1695487743
, "timeUseconds": 6467, "correlation_id": "3_2195560193@192.168.1.22", "protocolFamily": 2} | {"node": "13041664", "proto": "log"} | ciao ciao lorenzo
(1 row)

homer_data=# select protocol_header from hep_proto_1_call where sid='3_2195560193@192.168.1.22' limit 1;
                                                                                                                          protocol_header                                                                                                      

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
 {"dstIp": "192.168.1.180", "srcIp": "79.17.80.90", "dstPort": 5070, "srcPort": 5060, "protocol": 17, "captureId": "200", "payloadType": 1, "timeSeconds": 1695484999, "timeUseconds": 634671, "correlation_id": "3_2195560193@192.168.1.22", "
protocolFamily": 2}
(1 row)

[gmaruzz]

yep, they match

[lmangani]

The default mapping should then resolve the logs into the SIP session automatically. Are you sure the timestamps match?

Here's the default correlation for calls, as you can see the very first step takes the callid and lookup the type 100 logs:

[
    {
        "source_field": "data_header.callid",
        "lookup_id": 100,
        "lookup_profile": "default",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.callid",
        "lookup_id": 5,
        "lookup_profile": "default",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "protocol_header.correlation_id",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "sid",
        "lookup_range": [
            -300,
            200
        ]
    },
    {
        "source_field": "data_header.callid",
        "lookup_id": 1,
        "lookup_profile": "call",
        "lookup_field": "data_header->>'callid'",
        "lookup_range": [
            -300,
            200
        ],
        "input_function_js": "var returnData=[]; for (var i = 0; i < data.length; i++) { returnData.push(data[i]+'_b2b-1'); }; returnData;"
    }
]

[gmaruzz]

I see I got this error in js console (address redacted):

polyfills.39594fe71d4c206db379.js:1 POST https://123.4.5.6:5030/api/v3/agent/search/1dc62be9-bfab-4d71-a18e-1480205821c3/cdr 400 (Bad Request)

maybe this is the error that do not display the hep log related things?

[gmaruzz]

YEP, problem was the timestamps!

Thanks a lot Lorenzo!!!!! (as always!!) <3

[lmangani]

No this error seems unrelated. I can reproduce it when looking for cdrs but I still have logs.

Inspect the log request and check what's happening with it:

[lmangani]

Ah, I'm glad that makes sense now! Much love Giovanni!