Difficulty configuring HEPSUB for RTPEngine

[spacetourist]

I'm working on an implementation for call tracing based on two external platforms - opensips and RTPEngine, a common setup I believe. First my servers:

192.168.181.241 - Homer server (docker hom7-prom-all, last updated December 1st) 192.168.180.201 - RTPEngine server (also running HEPSUB via https://github.com/sipcapture/hepsub-rtpengine)

My ambition is to have all SIP collected and to implement a system which invokes RTP recording on a minority of calls, for those calls where RTP has been recorded I want to use HEPSUB subscription to trigger the retrieval from the RTP server where the capture occurs and to make this downloadable for the UI user allowing review of the SIP and collection of the PCAPs via the browser.

The OpenSIPs component is complete and functioning - I use external REST to decide when to record SIP and/or RTP for a dialog and this is successfully triggering the recording on the RTP server and pushing all SIP and logs into Homer - all working great. As a minor point I needed to use the homer5 syntax as shown in the params below, this isn't problematic for me but with v7 the data didn't appear.

 modparam("proto_hep", "hep_id", "[hep_dst] 192.168.181.241:9060; transport=udp; version=3")
 modparam("proto_hep", "homer5_on", 1)
 modparam("proto_hep", "hep_capture_id", 1)

I use a stock installation of Homer UI at this point and the SIP is easy to search and navigate - brilliant. Next I'll describe the pain point with the HEPSUB setup. Firewalls are open for all ports involved, I have performed traces that confirm no request is issued towards the hepsub agent from Homer.

The config for hepsub-rtpengine is shown below - I've struggled to decipher the purpose of all the values so want to be clear that these are best guesses only - in particular I'm not clear of the significance/use of gid/type:

{
  backend: "http://192.168.181.241:9080/api/v3/agent/subscribe",
  token: "LuLKnyOqCHdLioGpmBIvwuJWZmVCVOLgyINMitnRnFxXZQFSaOevpQUOuViaGGtcfaWOAdNbXXmRvaoL",
  rtpengine: {
    "path" : "/var/spool/rtpengine",
        "download" : true,
        "insecure" : true
  },
  service: {
    "uuid": "7cfa86a6-367e-4138-9941-6f4141b9110c",
    "host":"192.168.180.201",
    "port": 18088,
    "protocol": "http",
    "path": "/get",
    "type": "pcap;download",
    "ttl": 300,
    "node": "HR-P-RTP-1",
    "gid": 10
  },
  "debug": true
};

The HEPSUB agent exposes an API with path /get/:id which expects id to be either pcap OR download, not clear what logic there is in the Homer App whcih may or may not make this work however at this point I'm not seeing any requests so I'd like to tackle that first, just trying to be clear.

When starting the agent the token is accepted by Homer and a session is started, visible via the UI and in the database as shown below:

homer_config=# select * from agent_location_session;
 id  |                 guid                 | gid |      host       | port  | protocol | path |    node    |     type      |          create_date          |          expire_date          | active 
-----+--------------------------------------+-----+-----------------+-------+----------+------+------------+---------------+-------------------------------+-------------------------------+--------
 479 | aac3d1e0-7255-11ed-9767-19b080ed87f8 |  10 | 192.168.180.201 | 18088 | http     | /get | HR-P-RTP-1 | pcap;download | 2022-12-02 15:39:48.442856+00 | 2022-12-02 15:44:48.442857+00 |      1

With this registration active I am expecting to see requests being placed to the agent any time I click on a call on the UI which provide the call ID and check if the agent has the file, not sure if this is how it works? Also if that did work, I'm not clear on how to get the results to appear on the UI - I'd like a RTP PCAP download button to be added when its detected a recording - I believe I've seen similar implementations with additional UI objects whilst working on this.

I won't post my attempts at setting up mappings/correlations for this, these have all be stabs in the dark and haven't achieved anything that I can see so I'm hoping someone who is more familiar with the platform can request the specific/relevant parts.

I hope that all makes sense, I've run short on ideas and am really keen to get this moving again so will appreciate any input! Thanks for reading.

[spacetourist]

Thanks @adubovikov - I'm happy to crack on and submit more patches as required however I'm not super familiar with Docker and wondered if you could help me to understand how I can best test my changes without somehow creating my own images for homer-app? Do i simply stop that container and run a copy manually?

[adubovikov]

you can just compile it and run as normal app:

homer-app - run on port 8080

and in homer-ui, run it as

npm install

and after

npm run webdev

and all request api will go to port 127.0.0.1:8080 (your homer-app) and on port 80 you will have nodejs UI

https://github.com/sipcapture/homer-ui/blob/master/src/environments/environment.dev.ts#L8

in this case you can test, patch UI and go app without rebuild docker image

[lmangani]

I've added a docker package to the repo automatically generated at merge time: https://github.com/orgs/sipcapture/packages

The latest should include all changes and provide a replicable method to build your own containers straight off github

[spacetourist]

That's super helpful, thanks @lmangani

I'm working on this in production (all UDP) so I've simply modified the compose file for homer7-docker to use ghcr.io/sipcapture/homer-app:1.4.42 instead of sipcapture/webapp and its working great.

Am I correct that this is now part of the pipeline such that any new version tags will automatically generate a new container image that can be used prior to the docker hub version being updated?

[lmangani]

@spacetourist yes, this is now permanently part of the pipeline and will extend to all the other components shortly. The version tags + latest will be updated, too and eventually the Dockerhub images will also be pushed from this Action.

[spacetourist]

Hi @adubovikov and @lmangani - I've been familiarising myself with the UI code base and wanted to open a discussion about how I might be able to handle multiple agents, in my case only one of which may have the relevant data. I've put together a draft PR to illustrate here:

https://github.com/sipcapture/homer-ui/pull/607

Firstly I am keen not to start implementing changes which break other implementations, please shout if I'm going about this incorrectly!

From a practical standpoint after the recent patches were applied I've discovered that the lookup is only being performed against a single agent subscriber. My PR attempts to implement a "stop on first match" policy which iterates all the subscribers and performs sync requests to the getHepsubElements() API. The intention is to modify the HEPSUB subscriber service (running on my RTPEngine servers) to return 404 when the session was not found locally. This should mean that I can iterate over the agents and try each until one or none have found the data I'm looking for.

I have to head out for some family festive fun :christmas_tree: so can't engage on this immediately however I am keen to get some feedback, hopefully the PR makes sense.

Thanks!

[spacetourist]

Hi both, I see the comment from @lmangani on the draft PR - wondering if I should halt work if support for multiple agents can be pulled in from another project? I'm keen to get something operational but don't want to get in the way if this is something that can be ported in the short term. Let me know what you think about the suggestion @adubovikov and whether I can contribute to a solution here.

[adubovikov]

@spacetourist we can port, but unfortunately the guys are busy right now and it can be done next week. If you are feel good and familiar with the code, go ahead and make a pr with all changes, we will be more than happy to merge it :-)

[spacetourist]

Hi @adubovikov , @lmangani

I've been working on this and finally have a functional implementation where the logs dialog will now seek for data for all HEPSUB agents indicating that they provide pcap's. When found the panel appears and a download button is available - current layout shown below:

Format/style changes are only to demonstrate the basic functionality - this can be easily refined and I'm open to ideas here.

There are a few changes I'm still looking into however I've issued PR's for review at the current state:

https://github.com/sipcapture/homer-ui/pull/608 https://github.com/sipcapture/homer-app/pull/503

One further change I'm looking into is performing the lookups only against HEPSUB subscribers which are subscribed from an IP included in the source IP list - for my setup all subscribers run locally on the RTPEngine servers and it makes sense that I only need to perform lookups against those agents who appear in the media path as the others will never have the file. Something like that seems to drift a bit away from a general solution so please advise if I'd be better keeping that locally rather than issuing a PR for the project.

Similarly to the above if anything looks inflexible/wrong about the PR's above lets discuss and see if we can find a solution that works for the project and for my needs.

[spacetourist]

One change I wanted to highlight is: tab-hepsub.component.ts

The new method getRequest() is currently restructuring the basic data needed by the agent/search method to allow it to integrate cleanly with the DoSearchByPost() method in homer-app which previously used a hack of sorts to get the request out to the agent.

This was largely a result of my approach as I was going in blind and having to hack about to understand how the system was working. I'm not clear if there is a cleaner way to do this but I'm certainly open to suggestions as its currently adding data thats not required but if its useful to have consistent requests then perhaps its supposed to look like this!

[lmangani]

Thank you so much @spacetourist for taking the time to work on this! We'll review the changes as soon as possible!

[spacetourist]

No problem, I've enjoyed getting into the project. Note that the HEPSUB agent for RTPEngine required some reworking, I can issue a PR there next week. I believe the pcap path has not been shared and i'm looking it up again on download

[spacetourist]

I've added the HEPSUB changes to a further PR https://github.com/sipcapture/hepsub-rtpengine/pull/4

As per the above I'm happy to re-implement any of these if I'm drifting away from spec, just let me know of any issues.

Thanks

[adubovikov]

Meged. Thank you! To merge https://github.com/sipcapture/homer-ui/pull/608 , you have to remove draft

[spacetourist]

thanks, i've removed the draft labels and appreciate the merge!

[adubovikov]

done! Thank you so much!

[spacetourist]

excellent, theres one more at https://github.com/sipcapture/homer-app/pull/503 thats needed to get it all humming

[adubovikov]

done

[lmangani]

Release is ready: https://github.com/sipcapture/homer/pkgs/container/homer-app

[lmangani]

Is this thread resolved? Thanks @spacetourist for joining the contributor ranks, let us know if everything checks out and welcome to the hep bunch!

[spacetourist]

Yes i think so, thank you so much for all your assistance - hopefully I will find opportunities to make more contributions in the future!