New protocol support: SIP

[tsg]

Split from elastic/beats#22.

[kputland]

+1

[johntdyer]

:+1:

[marcelog]

+1

[sinkovsky]

+1

[donckers]

+1

[majofi]

+1

[crickes]

+1 for me too

[lukebeer]

I know SIP & RTP very well but not Go. If someone is willing to team up & be on IM to lend assistance with the language. I'm game.

[tsg]

Hi, @lukebeer, I'd be happy to pair program on that. Shoot me an email at tudor@elastic.co. Btw, we now have a new protocol dev guide: https://www.elastic.co/guide/en/beats/packetbeat/current/_developer_guide_adding_a_new_protocol.html

[crickes]

Hi, I thought about having a crack at this also but as the majority of SIP clients and all RTP use UDP instead of TCP, don't we have to wait until UDP is supported by PacketBeat?

[tsg]

@crickes UDP is supported now :-). See the DNS protocol implementation for an example.

[lukebeer]

Just to tac on, depending on the network topology, particularly when the UAs are within the same network, SDP will typically negotiate a direct media path between the endpoints involved separately to SIP signalling so capturing the RTP stream won't be possible without anchoring the media through a server, SBC of some kind or span port switch configuration.

If media is available to the monitor, the best way to store RTP detail will be with the RTCP reports generated by each end point, if supported.

In short, a SIP module is a great starting point for VoIP support so let's get this nailed and worry about RTP after. Unless of course a volunteer is willing :)

Hi, I thought about having a crack at this also but as the majority of SIP clients and all RTP use UDP instead of TCP, don't we have to wait until UDP is supported by PacketBeat?

— Reply to this email directly or view it on GitHub https://github.com/elastic/packetbeat/issues/152#issuecomment-138465383.

[tsg]

@lukebeer +1, we should start with SIP. I think @crickes meant that SIP is also often run over UDP.

[lukebeer]

Agreed and this brings up a good point, how to track calls - TCP streams give us the opportunity to just follow the id however with UDP it'll be call id/branches/cseq's.

That said, should calls be correlated or just treat each message as an individual? Thoughts?

Perhaps it's time to turn to the RFC....

mail@luke.beer

On 8 September 2015 at 17:48, Tudor Golubenco notifications@github.com wrote:

@lukebeer https://github.com/lukebeer +1, we should start with SIP. I think @crickes https://github.com/crickes meant that SIP is also often run over UDP.

— Reply to this email directly or view it on GitHub https://github.com/elastic/packetbeat/issues/152#issuecomment-138629825.

[crickes]

Hi,

With regards to RTP, they are just flows like anything else. The issue gets more interesting if you want to analyse quality of service. RTCP does give some metrics regarding packets expected and received to give a measure of packet loss, but the extended receiver report has a lot of other QoS metrics in it, but not all endpoints support this.

With @lukebeer comment about tracking SIP calls. In my opinion, it would be better to track SIP dialogues, rather then complete calls. A call can last anywhere from a few seconds to several hours, and tracking the call states purely in the probe is potentially very tricky and very memory intensive. It would be similar to tracking a users visit to a website based on their browser session ID. Perhaps a better approach is to track the individual dialogues, Request to Response but make a note of the Call ID, just as you would with a session ID in tracking a HTTP GET. You could then use the power of Elasticsearch to create an entity centric (Call / session) index to see the call states, rather than trying to track them in the packetbeat probe.

I'm not sure how PacketBeat works as yet, I've just started looking at it, but if you were to wait until the call was completed before it updates ES with the call details, you could potentially be waiting a long time (the call duration) before you were aware of the call at all. Tracking the initial INVITE to response, makes you aware of the call attempt immediately and then based on the response you can infer whether the call is in progress or not. You would then add each additional dialogue to the database using the call-ID to relate the dialogues to the call.

[crickes]

A bit of background information, my colleague and I have already tested sending individual SIP packets into ES and have been able to see rates of message types, ie, 500 INVITES / Sec.. The scale of the solution we are creating makes this individual packet counting technique unscalable (several 10,000s on packets per second being sent to an ES cluster). Using packetbeat to track the dialogues would cut down on the number of inserts into the database and also provide some extra metrics like time to respond, INVITE -> 100 (server response time), or INVITE -> 18x (Post dial delay) for example.

I personally have never heard of 'GO' before reading this thread, but I am very familiar with SIP and RTP, and willing to learn 'GO' and help in the development or at least testing of this module. I work for one of the UK's largest providers of SIP trunks and I can push LOTS of real SIP traffic into a test box and see how it performs under heavy load.

[lukebeer]

@crickes - perfect. Track the dialogs then correlate later, hopefully it'll stand up to the throughput. My concern was specifically how we'd deal with long duration calls. To be honest, it didn't occur to me to just track the dialog, that's a much better approach. Input appreciated.

Also - what a coincidence! I've just signed the papers and will be joining your workforce on the 22nd of October, greetings!

[lukebeer]

Had a thought - a while back I needed a way of handling an insane transaction rate without loss. I ended up using libnfqueue to distribute the traffic amongst multiple nodes running an open source message broker. Each node in this pool acted as a buffer by replaying the packets at a rate the destination could withstand. We could use the same approach here if, when required.

[crickes]

@lukebeer Look forward to working with you on this and potentially other things then! Look me up on linked in.

[urso]

Just some input:

Currently packetbeat has no support for SCTP. Check which transport protocols you will need for your use case. If you can do without SCTP just go ahead.

For how long do you want track the dialog? When and how often do you want to publish dialog states? A dialog is identified by SIP call id, from tag and to tag. For transactions use the cseq in addition. When working on dialogs be aware that transaction (different cseq numbers) might be active at about the same time...

About SIP call leg (hop-by-hop): People sometimes mix TCP and UDP on the very same call leg. Don't trust port numbers, transport protocol, IPs, Mac addresses... there are various combinations of how ethernet, ip, vlan, sip call id are used to make up a leg. Be flexible in configuring the correlation of messages into transactions and or call legs. When handling transactions be prepared to see request via TCP and response via UDP (just saying). Also be aware of Non-RR.

At the moment packetbeat is supposed to be mostly single-threaded. For now you have to solve load-balancing somewhat yourself.

You might want to treat packetbeat mostly as a probe.Consider concentrating on publishing transactions first instead of building a full state machine on dialogs. Transactions are relatively short-lived and you won't tax memory usage that much. Having transactions you will already get registration support. Building call leg state machines based on transactions might simplify the task of handling message reorderings and other weird stuff.

For building up dialogs maybe you can do it in ES based on transactions, maybe some logstash plugin collecting transactions from multiple probes. Plus how often/when do you want to publish call state updates?

About RTP/RTCP: you trust RTCP reports? Or do you plan on collecting similar stats for RTP.

[kputland]

Some thoughts,

• Transactions are better to have sooner rather than later. Think watcher based fraud detection routines that can run on say receipt of some INVITE/second to high risk destinations.
• Call legs and dialogs can be correlated later
• Correlation can be tricky across SBCs if the SBCs doing topology hiding or other HMR that changes callid, or tags.

--Karl

Karl Putland Senior Engineer SimpleSignal Anywhere: 303-242-8608 http://www.simplesignal.com/explainer_video.php

On Tue, Sep 8, 2015 at 4:06 PM, Steffen Siering notifications@github.com wrote:

Just some input:

Currently packetbeat had no support for SCTP. Check which transport protocols you will need for your use case. If you can do without SCTP just go ahead.

For how long do you want track the dialog? When and how often do you want to publish dialog states? A dialog is identified by SIP call id, from tag and to tag. For transactions use the cseq in addition. When working on dialogs be aware that transaction (different cseq numbers) might be active at about the same time...

About SIP call leg (hop-by-hop): People sometimes mix TCP and UDP on the very same call leg. Don't trust port numbers, transport protocol, IPs, Mac addresses... there are various combinations of how ethernet, ip, vlan, sip call id are used to make up a leg. Be flexible in configuring the correlation of messages into transactions and or call legs. When handling transactions be prepared to see request via TCP and response via UDP (just saying). Also be aware of Non-RR.

At the moment packetbeat is supposed to be mostly single-threaded. For now you have to solve load-balancing somewhat yourself.

You might want to treat packetbeat mostly as a probe.Consider concentrating on publishing transactions first instead of building a full state machine on dialogs. Transactions are relatively short-lived and you won't tax memory usage that much. Having transactions you will already get registration support. Building call leg state machines based on transactions might simplify the task of handling message reorderings and other weird stuff.

For building up dialogs maybe you can do it in ES based on transactions, maybe some logstash plugin collecting transactions from multiple probes. Plus how often/when do you want to publish call state updates?

About RTP/RTCP: you trust RTCP reports? Or do you plan on collecting similar stats for RTP.

— Reply to this email directly or view it on GitHub https://github.com/elastic/packetbeat/issues/152#issuecomment-138717626.

[claygorman]

+1

[howken]

+1

[effeks]

+1

[ESamir]

+1

[damianpfister]

+1

[srpatatas]

+1

[yanou75pro]

+1

[fcastello]

Any progress in adding SIP to packetbeat? Is there a way I can test what you were working?

[darrenliddell]

+1

[chaunbot]

+1

[Jayranbutong]

so ,how is going

[Haphil]

+1

[sfrique]

+1

[ekoome]

Latest on this?

[kisienya]

+1

[bigtoerag]

@gerardseeto has forked packetbeat and has some good ideas on a way forward but needs help with learning GOlang.

Can anyone with GO skills contact him and see if its easy to progress?

[kisienya]

Try these resources:

https://tour.golang.org/welcome/1 https://gobyexample.com/

Thanks

[kisienya]

One more...https://www.golang-book.com/

[tintoy]

Hi @bigtoerag.

I work for the same organisation as @gerardseeto and will have a chat to him to see if I can help (my Go skills aren't particularly l33t but they're serviceable).

[fatihrd]

+1

[jatsmulator]

We already implement SIP and MSRP protocol plug-ins with Packetbeats, SIP and MSRP packets can be recovered and shown in ELK.

[bigtoerag]

Not sure what you mean as I checked and its not listed yet. https://www.elastic.co/products/beats/packetbeat

[monicasarbu]

@jatsmulator Happy to hear you already added support for SIP and MSRP in Packetbeat. Would be great if you can share it with the community and create a PR upstream.

[kisienya]

@jatsmulator Where can we get this since its not listed on https://www.elastic.co/products/beats/packetbeat

[jatsmulator]

Hi Kisienya:

Happy new year! Yes, I am starting create a PR, and add SIP and MSRP support into Packetbeat. Please wait for a while.

2017-02-06 16:18 GMT+08:00 kisienya notifications@github.com:

@jatsmulator https://github.com/jatsmulator Where can we get this since its not listed on https://www.elastic.co/products/beats/packetbeat

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/elastic/beats/issues/152#issuecomment-277611836, or mute the thread https://github.com/notifications/unsubscribe-auth/APOgdeKFRigsQGtnCnN1-GUAdBeXclRsks5rZtdugaJpZM4E01dj .

[kisienya]

@jatsmulator many thanks for that...

[janprabu]

Hi,

Could you share that plugin code. We wants to try that.

Thanks in advance.

[tsg]

@jatsmulator Hey, any progress on that? Feel free to open a PR even if not all the functionality is in there, we are glad to help with that.

[smokersonly]

any estimate when this would be available?