AudioFork does not work on Asterisk 19

kobaz commented 2 years ago

AudioFork leaves dangling channel references after channel hangup

vbox-markm-x64*CLI> core show channels
Channel              Location             State   Application(Data)
0 active channels
0 active calls
4 calls processed
[2022-05-04 14:52:37.486-0400]     -- Executing [1234@cos_foo-c10000:1] Answer("PJSIP/c10000-102-00000004", "") in new stack
[2022-05-04 14:52:37.487-0400]        > 0x7f424c1a7bc0 -- Strict RTP learning after remote address set to: 192.168.50.206:2252
[2022-05-04 14:52:37.587-0400]        > 0x7f424c1a7bc0 -- Strict RTP switching to RTP target address 192.168.50.206:2252 as source
[2022-05-04 14:52:37.587-0400]     -- Executing [1234@cos_foo-c10000:2] Verbose("PJSIP/c10000-102-00000004", "starting audio fork") in new stack
[2022-05-04 14:52:37.587-0400] starting audio fork
[2022-05-04 14:52:37.587-0400]     -- Executing [1234@cos_foo-c10000:3] AudioFork("PJSIP/c10000-102-00000004", "ws://127.0.0.1:8888/capture?op=inbound") in new stack
[2022-05-04 14:52:37.587-0400]   == setting wsserver to ws://127.0.0.1:8888/capture?op=inbound
[2022-05-04 14:52:37.587-0400]   == setting direction to 2
[2022-05-04 14:52:37.587-0400]     -- Executing [1234@cos_foo-c10000:4] Verbose("PJSIP/c10000-102-00000004", "audio fork was started continuing call..") in new stack
[2022-05-04 14:52:37.587-0400] audio fork was started continuing call..
[2022-05-04 14:52:37.587-0400]     -- Executing [1234@cos_foo-c10000:5] Playback("PJSIP/c10000-102-00000004", "hello-world") in new stack
[2022-05-04 14:52:37.587-0400]     -- <PJSIP/c10000-102-00000004> Playing 'hello-world.ulaw' (language 'en')
[2022-05-04 14:52:37.602-0400]   == Connecting websocket server at ws://127.0.0.1:8888/capture?op=inbound
[2022-05-04 14:52:37.602-0400]   == Creating WS without TLS
[2022-05-04 14:52:37.605-0400] WARNING[25108]: tcptls.c:656 ast_tcptls_client_start_timeout: Unable to connect websocket client to 127.0.0.1:8888: Connection refused
[2022-05-04 14:52:37.605-0400] ERROR[25108]: app_audiofork.c:473 audiofork_thread: Could not connect to websocket on audio form PJSIP/c10000-102-00000004
[2022-05-04 14:52:38.662-0400]     -- Executing [1234@cos_foo-c10000:6] MusicOnHold("PJSIP/c10000-102-00000004", ",300") in new stack
[2022-05-04 14:52:38.662-0400]     -- Started music on hold, class 'default', on channel 'PJSIP/c10000-102-00000004'
[2022-05-04 14:52:40.071-0400]     -- Stopped music on hold on PJSIP/c10000-102-00000004
[2022-05-04 14:52:40.071-0400]   == Spawn extension (cos_foo-c10000, 1234, 6) exited non-zero on 'PJSIP/c10000-102-00000004'
[2022-05-04 14:52:40.071-0400]     -- Executing [h@cos_foo-c10000:1] Set("PJSIP/c10000-102-00000004", "__DialedNumber=h") in new stack
[2022-05-04 14:52:40.071-0400]     -- Executing [h@cos_foo-c10000:2] Set("PJSIP/c10000-102-00000004", "__Tenant=foo-c10000") in new stack
[2022-05-04 14:52:40.071-0400]     -- Executing [h@cos_foo-c10000:3] Gosub("PJSIP/c10000-102-00000004", "Core_PreRoute,s,1") in new stack
[2022-05-04 14:52:40.071-0400]   == Spawn extension (cos_foo-c10000, h, 3) exited non-zero on 'PJSIP/c10000-102-00000004'
vbox-markm-x64*CLI> core show channels
Channel              Location             State   Application(Data)
PJSIP/c10000-102-000 h@cos_foo-c10000:3   Up      Gosub(Core_PreRoute,s,1)
0 active channels
0 active calls
5 calls processed
vbox-markm-x64*CLI> audiofork list PJSIP/c10000-102-00000004
No channel matching 'PJSIP/c10000-102-00000004' found.
vbox-markm-x64*CLI>

Also... Server-Side tcpdump (all data for port 8888):

14:01:45.874275 IP localhost.8888 > localhost.43284: Flags [S.], seq 3564456825, ack 3639784400, win 65483, options [mss 65495,sackOK,TS val 3944669654 ecr 39446696                                   le 7], length 0
E..<..@.@.<........."....uKy.........0.........
............
14:01:45.874282 IP localhost.43284 > localhost.8888: Flags [.], ack 1, win 512, options [nop,nop,TS val 3944669654 ecr 3944669654], length 0
E..4B0@.@............."......uKz.....(.....
........
14:01:45.874486 IP localhost.43284 > localhost.8888: Flags [P.], seq 1:202, ack 1, win 512, options [nop,nop,TS val 3944669654 ecr 3944669654], length 201
E...B1@.@............."......uKz...........
........GET /capture?op=inbound HTTP/1.1
Sec-WebSocket-Version: 13
Upgrade: websocket
Connection: Upgrade
Host: 127.0.0.1:8888
Sec-WebSocket-Key: PLCvXwAAAADJ8dpIAAAAAA==
Sec-WebSocket-Protocol: echo

14:01:45.874508 IP localhost.8888 > localhost.43284: Flags [.], ack 202, win 511, options [nop,nop,TS val 3944669654 ecr 3944669654], length 0
E..4..@.@..>........"....uKz.........(.....
........
14:01:45.877353 IP localhost.8888 > localhost.43284: Flags [P.], seq 1:128, ack 202, win 512, options [nop,nop,TS val 3944669657 ecr 3944669654], length 127
E.....@.@..........."....uKz...............
........HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: LZR976y3LNYYRc88w4w3vT8CBzw=

There is no more data... despite the call completing and going to MusicOnHold... the audio stream is never sent.

nadirhamid commented 2 years ago

@kobaz

Thanks for filing this issue.

I have yet to test the module with Asterisk 19. But a quick review could suggest that its config related and maybe something with the WebSocket server. In a nutshell, some older Websocket libraries are incompatible with the module and many changes made to address this issue. I am not sure if this is exactly the issue, but it is highly likely that something such as the WS library is causing this issue.

Just so I can understand this better: Can you please let me know what language and library you used to create the backend ?

kobaz commented 2 years ago

Hi,

It seemed to be a server-side issue. My first attempt was connecting to a php-websocket server which was not working. I switched to Net::WebSocket::Server which works great.

I had to fix a few issues with the code:

Websocket connection is not closed on channel hangup, leading to leak and server not knowing connection is finished
Dangling channel reference, leaving channels around after-hangup, leading to 'phantom channels' remaining after hangup
Basic re-connection support (Single attempt.. but really it should continually try and reconnect as long as the channel is up.. to a time limit)

I'll get a pull request going for you.

kobaz commented 2 years ago

Hi,

Also. I am a part-time developer for the Asterisk Project. I primarily maintain the AEL module. I am interested to get this audiofork module of yours officially added to the Asterisk Project.

Would you be willing to sign a (free) developer agreement with Digium/Sangoma so we can include this module in Asterisk?

Thanks!

nadirhamid commented 2 years ago

I am very open to this. Please send the details and I will have a look

kobaz commented 2 years ago

Hi Nadir,

Here you go: You'll need a new free account with the Asterisk Issue Tracker. https://signup.asterisk.org/signup

Then submit the developer agreement: https://issues.asterisk.org/jira/secure/DigiumLicense.jspa

You'll want to review this: https://www.asterisk.org/community/developers/

As well as the coding standards (the main thing being Asterisk uses tabs instead of spaces, so the code will have to be re-intended and change every 2 spaces to one tab) https://wiki.asterisk.org/wiki/display/AST/Coding+Guidelines

kobaz commented 2 years ago

A few additional things should be taken care of as well.

Add periodic re-connection support (and have it be configurable using parameters) Example: It should be possible to pass in: reconnection attempts: 5 (abort after this many attempts), or 0 for unlimited reconnection interval: 30 (number of seconds between reconnection attempts) reconnection buffer (in kilobytes) -- (If the call is hours and hours, and the connection is down for an extended period.. holding a large buffer of frames will lead to high memory usage. There will come a point in time where frames will need to be dropped/skipped)
Add Asterisk TestSuite tests https://wiki.asterisk.org/wiki/display/AST/Asterisk+Test+Suite+Documentation These are automated tests built into the codebase that would ensure that functionality is working correctly when the project is fully compiled

nadirhamid commented 2 years ago

@kobaz I will have a look at this over the next few days. Thanks!

nadirhamid commented 2 years ago

@kobaz Reviewed it. And agreement is done.

I'll be looking over the coding changes shortly. They shouldn't be too complex.

nadirhamid commented 2 years ago

Hello,

Just looking into this now, and I had a few quick observations.

Firstly, based on the code right now, a routine for reconnections would be enhancement but not critical to the overall functionality. I will consider this if you really think you need this, but it seems to be a small upgrade and not crucial to how it works. Right now, the script creates a connection to the WS server each time a new channel is created (or when the app is called), and there is one connection per channel.

If you want to add reconnection support, can you please let me know where you want to add it.

For example, are you suggesting that we try to reconnect before we create the first connection, or should the app try to reconnect if connections are lost ? I would need to know more aboout this.

Secondly, referring to the test suite, it doesnt seem like this will be so complex so I can look into it. I will have to study the Asterisk core and see how tests were done there, but integration wise, it can be added.

If you can get back to me about the reconnection part it would be much appreciated.

Thanks

kobaz commented 2 years ago

HI Nadirhamid,

For production-readyness and robust behavior there should be options to avoid data-loss, such as more than one reconnect, and possibly storing data on disk.

In terms of reconnection support... if the remote websocket server becomes unavailable, the audiofork module should keep trying to connect, with exponential timeouts. Up until an upper limit.

Example 1:- -WebSocket Server becomes unavailable (drops TCP connection) -AudioFork tries to reconnect immediately, fails -AudioFork tries to reconnect after 1 second, fails -AudioFork tries to reconnect after 2 seconds, fails -AudioFork tries to reconnect after 4 seconds, fails -AudioFork tries to reconnect after 8 seconds, fails -....Keep trying to reconnect until the timeout is hit (either audiofork.conf file, or an option to AudioFork() application)

Example Timeout: 120 seconds.
In this example, AudioFork will keep up to 120 seconds of audio in memory so that no data is lost.
When reconnecting, we'll keep streaming, continuing at the last frame we have successfully sent (same as now)

Bonus features (not required, but would be nice to have) -If WebSocket server is completely failed... store raw frames in asterisk spool directory on disk When WebSocket server returns, stream disk-recorded frames back to websocket server

If the call has ended, but the WebSocket server is still unavailable, keep the AudioFork running until the end of the reconnect timeout to see if maybe the WebSocket server will come back so we can finish the recording (Maybe use Channel AutoService?)

nadirhamid commented 2 years ago

@kobaz sorry been delayed with this. However I did create a branch and I am currently testing some of these changes.

I will keep you notified of progress. And as soon as there is a working version ready, I will update this thread.

In the meantime, if you have any other queries or need more info please let me know.

Thanks

assistyoubram commented 1 year ago

Just to add on this:

Older & possibly Certified AST is not sticking RFC6455; https://issues.asterisk.org/jira/browse/ASTERISK-28949

You need a recent version of Asterisk.

kobaz commented 1 year ago

Thanks for the updates. I'm going to work on getting this posted up on the reviewboard for official Commit.

Do you have any pending work to commit?

nadirhamid / asterisk-audiofork

AudioFork does not work on Asterisk 19 #14