search

open-webrtc-toolkit / owt-server

General server (streaming/conference/transcoding/anayltics) for OWT. (A.k.a. MediaServer)
https://01.org/open-webrtc-toolkit
Apache License 2.0
1.13k stars 369 forks source link

Transcoding Issue: Opus -> AAC #1064

Closed PonyHugger closed 3 years ago

PonyHugger commented 3 years ago

G'day!

We are currently experiencing a weird issue, when it comes to transcoding Opus to AAC. It happens, when publishing a HLS stream or recording a session (in h264/aac). The audio stream is always included in the recorded videos - it is just silent! We have multiple workers on a cluster and on all machines transcoding is stopping to work after some time. There is no information in the logs. We have to reboot the servers, in order to make transcoding functional again. We are using hardware encoding, but i'm not sure, if that makes a difference, as only the video feed is hw encoded. We also played a bit with different cluster strategies, like having the audio and recording agent on a single machine.

Any ideas?

Cheers!

starwarfan commented 3 years ago

G'day! Which branch are you using? Can you record video-only stream and audio-only stream to locate the issue`? When does it stop to work after recording start?

PonyHugger commented 3 years ago

Hi ChenLi, thanks for your quick response! We are using the latest release from the 14th of April. (v5.0) It is hard to test, whether video-only and/or audio-only is working, as this issue is appearing after some time on single workers, on our productive environment. Once we notice, that a worker is not working correctly, we have to restart all workers, as there is no information in the logs, which one is failing... It is failing right from the start, on recording and HLS streams.

starwarfan commented 3 years ago

If there were no information in the recording logs, were there any warnings/errors in conference logs?

PonyHugger commented 3 years ago

Unfortunately, there are no warning/errors or whatsoever... We had that problem two days ago. We've checked all logs, but there was no indication for a problem. Then we've deleted all logs, to trigger the recording (with no sound) and check the logs again. But the new logs were also fine... When we ran into this issue for the first time, a couple of weeks ago, we found this message in one of the logs: [libfdk_aac @ 0x3789840] 2 frames left in the queue on closing ...but as this message did not pop up again, i assume that this has nothing to do with the current problem. Another point is, that we have to reboot the hole server, to get the audio stream back while recording or streaming a HLS feed. We don't have that issue on our dev enviroment, which is not using hw acceleration. Maybe this is related to the Intel Media SDK?

Cheers!

starwarfan commented 3 years ago

I think Media SDK won't affect audio. Can you restart recording agent only to get the audio stream back?

PonyHugger commented 3 years ago

No, unfortunately not. We tried different things, but we have to reboot the whole machine, to get the audio stream back in hls streams and recordings - while the WebRTC stream is not affected.

starwarfan commented 3 years ago

To address this issue, could you edit "conference_agent/log4js_configuration.json", change the log level of "Conference" and "roomController" to "DEBUG", restart the server and collect logs when this problem occurs?

PonyHugger commented 3 years ago

Thanks for your quick reply! I just applied the changes on the log4js_configuration.json and get back to you, once the problem appears again. Thank you very much!

PonyHugger commented 3 years ago

It seems like that this issue was related to once specific machine. The problem hasn't appeared since setting up the whole machine again. Thanks for your support, @starwarfan!