Audio is rattling

[linhlq58]

@pedroSG94 Hi pedro, I found an issue that audio is rattling on some device. You can check this video: https://www.facebook.com/watch/live/?ref=watch_permalink&v=1002159117298243

The device that I'm using to livestream is Oppo Reno6. Everything is fine but the audio is not okay.

[pedroSG94]

Hello, Are you using AWS as server? This server seems that produce noise for some reason using few devices: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/issues/925

If you are not using AWS, which server are you using? Facebook?

[pedroSG94]

Also, can you specify me exact device model (Reno6 Pro or Reno6 5G) and audio/video configuration?

[linhlq58]

I'm using Facebook server. The device model is Reno6 5G (OPPO CPH2251).

[linhlq58]

@pedroSG94 Here is my audio/video configuration:

• Video: 1280 x 720, 30fps, bitrate 5000 * 1024
• Audio: bitrate 128 * 1024, sample rate 44100, isStereo true, echoCanceler and noiseSuppressor are false

[pedroSG94]

Another question. If you record a video localy using my library instead of stream. Do you have the same audio problem?

[agent10]

There are two possible reasons here: 1) Device doesn't support the sample rate and silently switches it to another sample rate. But inited AudioEncoder doesn't know about that. 2) PTS for audio is calculated in incorrect way using raw System.nanoTime() diff with previous value. Maybe it makes sense to calculate with buffer duration instead, like Android built-in screen recorder does it

[pedroSG94]

There are two possible reasons here:

1. Device doesn't support the sample rate and silently switches it to another sample rate. But inited AudioEncoder doesn't know about that.
2. PTS for audio is calculated in incorrect way using raw System.nanoTime() diff with previous value. Maybe it makes sense to calculate with buffer duration instead, like Android built-in screen recorder does it

I think that it is case 2. I can't test it because I haven Reno6 but I will implement it in a branch and share it to check if the problem is solved with that

[agent10]

In my case it fixed the all audio glitches I had. But unfortunately it leads to another problem that audio may become non synced with video due to delays/lags in android audio subsystem..

[pedroSG94]

I did a branch about it: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/tree/i956 Can you test it and let me know if it is working as expected? I will work on async problem after that.

[agent10]

@pedroSG94 I prepared two videos streamed on YouTube(it's important, e.g. Twitch handles pts differently). Both videos were taken from low-level Huawei DRA-LX9 device. The first video is master branch, latter is i956 branch. https://youtu.be/gvygZJeRI0Y?t=37 https://youtu.be/jiFLvLzSSK0?t=12

The second one sounds better for me.

[pedroSG94]

The second seems better and no sync problems. Do you know how to reproduce your problem with async? Which device I should use to reproduce both problems? (audio glitches and audio async after the fix)

[agent10]

You're right that there is no async issue but because there were no lags inside Android. I will try to reproduce async issue tomorrow. I faced Android lags(and async issue) while connecting/disconnecting BT or wired headset many many times(I bet there are other ways to reproduce it but I know only these). This leads to the dropped audio buffers inside Android(but we don't know about that) and incorrect pts will be set to the next received audio buffer.

Unfortunately, I can't tell you which devices you should use to reproduce it(( Some devices work fine, some don't. I suppose that main reason here is that the time diff between received buffers is not close to the buffer duration itself. May be you should use low-level devices with 2GB RAM or less or you can try to use software encoders as well(My devices I encounter audio glitches have 2GB) But I know exactly that on YouTube it can be reproduced easier. I have never faced with this issue on Twitch, as I mentioned before.

[agent10]

Hey again. I reproduced async issue easily. https://www.youtube.com/watch?v=8Q4y0gwfyxE In the beginning there are no delays but after I connected/disconnected wired headset(with mic) several times you can hear that we got delay.

Looks like I removed video by mistake(( have you got a chance to watch it? I'll take another one if you haven't and need it

[pedroSG94]

I had no time to watch it. Anyway, I did a commit with a method to re sync audio: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/commit/519c7d416d027aa8fcb76ec9fe5c0fbf968e0699 Can you check if it is working? You only need start stream with OpenGlView example, async audio and press switch camera button to resync audio. If it is working I can think in a way to execute it automatically

[biviel]

hi, @pedroSG94 , as you know I'm doing some experiments on RTMP and RTSP, developing on Ricoh Theta Z1 360 camera... I can confirm that this change in branch worked well. Streaming to youtube is good now for me at least, no sound rattling when streaming to youtube. That rattling for me was only hearable on youtube, on my own platform it was fine before too.

Nice work!

Is there an option to increase volume a bit in library?

Thanks, Laszlo

[pedroSG94]

hi, @pedroSG94 , as you know I'm doing some experiments on RTMP and RTSP, developing on Ricoh Theta Z1 360 camera... I can confirm that this change in branch worked well. Streaming to youtube is good now for me at least, no sound rattling when streaming to youtube. That rattling for me was only hearable on youtube, on my own platform it was fine before too.

Nice work!

It is working but seems that could produce audio async so for now it is not included into the library until I find a way to solve it (I'm not able to reproduce yet)

Is there an option to increase volume a bit in library?

No, but you can add a filter to the PCM buffer increasing the volume:

    rtmpCamera1.setCustomAudioEffect(new CustomAudioEffect() {
      @Override
      public byte[] process(byte[] pcmBuffer) {
        //return buffer modified. The buffer size result must be equal to original buffer
        return pcmBuffer;
      }
    });

I'm not sure about the algorithm but you can try get an idea in this post: https://stackoverflow.com/questions/58459337/how-to-change-the-volume-of-a-pcm-stream-without-floating-point

[biviel]

I will do more tests for sure, seems it's even device dependent...? I see can set audioeffect via microphonemanager too, right?

Is it ok to do in prepareaudio() , is it required to add customeffect before createmicrophpne or after?

      public boolean prepareAudio() {
        microphoneManager.createMicrophone(44100, false, false, false);
        microphoneManager.setCustomAudioEffect(new CustomAudioEffect() {
            @Override
            public byte[] process(byte[] pcmBuffer) {
                //return buffer modified. The buffer size result must be equal to original buffer
                return pcmBuffer;
            }
        });
        // If the argument is omitted, 128 * 1024, 44100, true, false, false
        return this.audioEncoder.prepareAudioEncoder(128 * 1024, 44100, false,1024);
    }

On this 360 camera, there is android 7.1.1 and API 25. I did a lot of live streaming tests with different Android devices and I noticed that when internet bandwidth drops down a bit under the required bitrate there were audio/video sync issues after the bandwidth is back again during the live stream. Is there a particular class in your library where this could be handeld most convenient way for borh RTMP and RTSP?

I also tried to setup a Raspberry PI4 to use as router and to provide internet to my camera via USB cable so actually the raspberry was connecting to a wifi router, a mesh network. I did this to be able to do fast roaming between different wifi routers as when the android driven camera was doing roaming it was causing the bandwidth to drop a bit below minimum required upload speed to maintain live stream bitrat when switched to antoher wifi in the mesh. At the end I built my own modified iwd to maintain speed during switches, which helped, but in some rare cases still the streaming library wasn't able to make sure that video and audio remains in sync once the switch to stronger wifi is done (roaming ended). I'm only sharing this because it may be very important even if frames are dropped, when roaming happens, that's not an issue, but video audio sync issue is a gap I couldn't eliminate. Is it possible that some king of "resync" would be possible in your libary? that's triggered if there are soem frames dropped or at other events? Detecting network bandwidth issue may be the best. I will come back soon about that othet "issue" about RTMP/RTSP streaming I was facing.

Thanks!

[pedroSG94]

Is it ok to do in prepareaudio() , is it required to add customeffect before createmicrophpne or after?

No matter if you do it before or after. Also, you can do it after start (basically any time).

The only way to handle with bandwidth issue in this library is the adaptative video bitrate that will reduce the frame drop: https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/wiki/Adaptative-video-bitrate#version-164

About audio/video sync. In this library it is configured in presentationTimeUs provided by MediaCodec.BufferInfo each time you receive a frame in encoders callback. I'm not doing anything in RTMP or RTSP modules. The only way to re sync is check video and audio ts and try keep both similar. If you can comfirm that both have a similar value, the problem are in network/server/player. The bandwidth issue shouldn't be a problem with audio/video sync (at least in my case I couldn't reproduce this problem). Also, keep in mind that not only this library is the reason of this audio/video async in few cases the problem is in server/player side. I know a server that have a sync problem with RTSP and also some times re start the player could solve the problem. Anyway, I need a good enviroment to test bandwidth issue (I need buy a good router to reproduce it) because I did tests forcing 3G on movil to get a cap of bandwidth but I can't reproduce continues spikes

[biviel]

hi, ok, I understand, will look into adaptive video bitrate soon, after I get first running RTMP and RTSP as required. I tested other camera/device to stream to same server and player, via same network to reproduce and there the video/audio sync when bandwidth dropped down was good after bandwidth was back. Audio and video synced again. I may have to look into this later again. Thanks!

[biviel]

Hi, I'm facing again with issues toward youtube, etc. "rattling sound" it may be related to other audio/video settings too and device dependant as mentioned before. Will let you know if I found soemthing.