Problem with using OpenSIPS 3.1 as HEP-client when using TCP+HEPv3

[Shkiperon]

I found a problem when I was trying to use OpenSIPS 3.1 as HEP-client in pair with heplify-server. If I use UDP+HEPv3 or TCP+HEPv3 - I can see the metrics in Grafana about SIP-traffic from Prometheus (the heplify-server is connected to Prometheus), but when I use TCP+HEPv3 - there is nothing in homer_data.hep_proto_1_call about calls. If I use UDP+HEPv3 for linking OpenSIPS 3.1 with heplify-server - everything works correct.

I was checked heplify-server side with tcpdump - in both cases traffic with information about calls is exist.

[lmangani]

@Shkiperon are those packets showing in Homer search? the Grafana part depends on the actual packets before their statistics

[Shkiperon]

If TCP+HEPv3 is not, it doesn't show. Because they are not in the database

[lmangani]

Could you please capture a PCAP with a few of those HEPv3 packets on either side for us to analyze and show the relevant tracing OpenSIPS 3.1 configuration? since the integration is quote elegant we can 99% adjust this with OpenSIPS settings.

[Shkiperon]

log_level=3
log_stderror=no
log_facility=LOG_LOCAL7

udp_workers=4
tcp_workers=4

auto_aliases=no

socket=udp:OPENSIPS_IP:5060
socket=tcp:OPENSIPS_IP:5060
socket=hep_tcp:OPENSIPS_IP:6061
socket=hep_udp:OPENSIPS_IP:6061

####### Modules Section ########

#set module path
mpath="/usr/local/lib64/opensips/modules/"

#### SIGNALING module
loadmodule "signaling.so"

#### StateLess module
loadmodule "sl.so"

#### Transaction Module
loadmodule "tm.so"
modparam("tm", "fr_timeout", 5)
modparam("tm", "fr_inv_timeout", 30)
modparam("tm", "restart_fr_on_each_reply", 0)
modparam("tm", "onreply_avp_mode", 1)

#### Record Route Module
loadmodule "rr.so"
/* do not append from tag to the RR (no need for this script) */
modparam("rr", "append_fromtag", 0)

#### MAX ForWarD module
loadmodule "maxfwd.so"

#### SIP MSG OPerationS module
loadmodule "sipmsgops.so"

#### FIFO Management Interface
loadmodule "mi_fifo.so"
modparam("mi_fifo", "fifo_name", "/tmp/opensips_fifo")
modparam("mi_fifo", "fifo_mode", 0666)

#### PostgreSQL module
loadmodule "db_postgres.so"
modparam("db_postgres", "exec_query_threshold", 60000)

#### DIALOG module
loadmodule "dialog.so"
modparam("dialog", "dlg_match_mode", 1)
modparam("dialog", "default_timeout", 21600)  # 6 hours timeout
modparam("dialog", "db_mode", 2)
modparam("dialog", "db_url",
    "CONNECTION_STRING")

#### RTPEngine module
loadmodule "rtpengine.so"
modparam("rtpengine", "db_url",
    "CONNECTION_STRING")

#### TRACER and HEP modules
loadmodule "proto_hep.so"
modparam("proto_hep", "hep_capture_id", CAPTURE_ID)
modparam("proto_hep", "homer5_on", 1)
modparam("proto_hep", "hep_id", "[hid]HOMER_DNS_NAME:9060;transport=udp;version=3")

loadmodule "tracer.so"
modparam("tracer", "trace_id", "[tid]uri=hep:hid")

#### UDP protocol
loadmodule "proto_udp.so"

#### TCP protocol
loadmodule "proto_tcp.so"
modparam("proto_tcp", "trace_destination", "hid")
modparam("proto_tcp", "trace_on", 1)

####### Routing Logic ########

route{
    route(to_homer);
    $var(trace_id) = "tid";

    force_rport();
    if (!mf_process_maxfwd_header(10)) {
        send_reply(483,"Too Many Hops");
        exit;
    }

    if (has_totag()) {
        #### BLOCK OF LOGIC
    }
    route(relay);
}

route[relay] {
    # for INVITEs enable some additional helper routes
    if (is_method("INVITE")) {
        #### BLOCK OF LOGIC
    }

    if (!t_relay()) {
        send_reply(500,"Internal Error");
    }
    exit;
}

branch_route[per_branch_ops] {
    xlog("new branch at $ru\n");
}

failure_route[missed_call] {
    if (t_was_cancelled()) {
        exit;
    }
}

route[to_homer] {
    $var(trace_id) = "tid";
    $var(trace_type) = NULL;

    if (!has_totag()) {
        if (is_method("INVITE")) {
            $var(trace_type) = "dialog";
        } else if (!is_method("CANCEL")) {
            $var(trace_type) = "transaction";
        }
    } else if (is_method("SUBSCRIBE|NOTIFY")) {
        $var(trace_type) = "transaction";
    } else {
        $var(trace_type) = NULL;
    }

    switch ($var(trace_type)) {
    case "dialog":
        trace("$var(trace_id)", "d", "sip|xlog|rest");
        break;
    case "transaction":
        trace("$var(trace_id)", "t", "sip|xlog");
        break;
    case "message":
        trace("$var(trace_id)", "m", "sip|xlog");
        break;
    }
}

If I change in this conf-file to "tcp" transport for hep - it will not be working. Some logic has been removed, but it shouldn't be needed for analysis

About PCAPs - I can show only the screenshots from Wireshark (I'm preparing them now).

[Shkiperon]

[lmangani]

Thanks for the detailing, but we need to see the actual HEPv3 packet dissected to understand what might be wrong. Could you install the HEP plugin for wireshark and attach the dissected headers, expanedd?

[Shkiperon]

[Shkiperon]

It looks like it was my mistake in setting up the heplify-server. I fixed it. Before:

HEPAddr = "0.0.0.0:9060"
HEPTLSAddr = "0.0.0.0:9060"

After:

HEPAddr = "0.0.0.0:9060"
HEPTCPAddr = "0.0.0.0:9060"