Two RTMPs are merged, and the video after merging is stuck one minute later

[libenli]

The two channels of rtmp, including audio and video, are combined and then pushed to rtmp, which is played using VLC. It is found that the video will be stuck after more than a minute, but there is mixed audio. What is the reason? Thank you

Here is the code:

String filterStrVideo = "color=c=black:size=1280*640 [base];[0:v]scale=640:360 [upperleft]; [1:v]scale=640:360 [upperright];[base][upperleft] overlay=0:80[tmp1];[tmp1][upperright] overlay=640:80[v]";
        String filterStrAudio = "[0:a][1:a]amix=inputs=2:duration=longest[a]";

        avutil.av_log_set_level(avutil.AV_LOG_INFO);
        FFmpegLogCallback.set();
        FFmpegFrameGrabber stream0 = new FFmpegFrameGrabber(pullUrl);
        stream0.start();

        FFmpegFrameGrabber stream1 = new FFmpegFrameGrabber(pullUrl2);
        stream1.start();

        int frameRate = (int)stream0.getVideoFrameRate();
        int frameWidth = stream0.getImageWidth();
        int frameHeight = stream0.getImageHeight();
        int audioChannels = stream0.getAudioChannels();

        FFmpegFrameFilter filter = new FFmpegFrameFilter(filterStrVideo, filterStrAudio, frameWidth, frameHeight,audioChannels);
//        FFmpegFrameFilter filter = new FFmpegFrameFilter(filterStrVideo, frameWidth, frameHeight);
        filter.setVideoInputs(2);
        filter.setAudioInputs(2);
        filter.setPixelFormat(stream0.getPixelFormat());
        filter.setSampleFormat(stream0.getSampleFormat());
        filter.start();

        FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(SRS_PUSH_ADDRESS,
                frameWidth, frameHeight,
                audioChannels);

        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
        recorder.setFormat("flv");
        recorder.setFrameRate(frameRate);
        recorder.setGopSize(frameRate*2);
        recorder.setAudioChannels(audioChannels);

        recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
        recorder.setSampleRate(44100);
        recorder.setAudioOption("crf", "0");
        recorder.setAudioQuality(0);

        recorder.start();

        Frame frame0, frame1, frame, frameAudio = null;
        frame0 = stream0.grab();
        frame1 = stream1.grab();
        while ((frame0 = stream0.grabFrame()) != null && (frame1 = stream1.grabFrame()) != null) {
            filter.push(0, frame0);
            filter.push(1, frame1);

            if ((frame = filter.pull()) != null) {
                recorder.record(frame, filter.getPixelFormat());
            }
        }

        recorder.close();
        filter.close();
        stream0.close();
        stream1.close();

[saudet]

The timestamps are probably wrong. You'll need to make sure they are correct.

[libenli]

The timestamps are probably wrong. You'll need to make sure they are correct.

Thanks，I try to add setTimestamp() refer to other issue, It seems to work，the video mixed will not be stuck after more than a minute； But as the playing time goes on, the time delay is getting bigger and bigger, from the first 6 seconds to more than 30 seconds, and the audio and video will get stuck every 10 seconds after five minutes. The playback will resume again after using VLC. Is the time stamp I set incorrect?

Here is the part of code which I changed:

while (true) {
            frame0 = stream0.grabFrame();
            frame1 = stream1.grabFrame();

            if (frame0 != null) {
                filter.push(0, frame0);
            }

            if (frame1 != null) {
                filter.push(1, frame1);
            }
            if (frame0 == null || frame1 == null) {
                if (err_index > 3) {
                    break;
                }
                err_index++;
                continue;
            }

            if ((frame = filter.pull()) != null) {
                long rt = recorder.getTimestamp();
                if (frame.timestamp > rt) {
                    recorder.setTimestamp(frame.timestamp);
                }

                recorder.record(frame, filter.getPixelFormat());
            }

        }

[saudet]

You'll need to make sure the timestamps of all the frames stay close to each other, usually something like within a second.

[libenli]

Is it the original input frame or the filtered frame

You'll need to make sure the timestamps of all the frames stay close to each other, usually something like within a second.

Is it the original input frame or the filtered frame？Then compare it with the timestamp of the Record Frame？

[saudet]

Is it the original input frame or the filtered frame？Then compare it with the timestamp of the Record Frame？

I did said all the frames, input, output, filtered, all the frames, all of them.

[libenli]

I printed the timestamps of all frames and reset the video timestamps according to your instructions, but how do I set the audio? Now the result is that if the audio filter is shielded in the filter, the playback is normal, but if the audio filter is turned on at the same time, the video after merging will still be stuck after 3 minutes. Please take a look. Thanks a lot.

Here is the code:

while ((frame0 = stream0.grabFrame()) != null && (frame1 = stream1.grabFrame()) != null) {
            filter.push(0, frame0);
            filter.push(1, frame1);
            long t0 = stream0.getTimestamp();
            long t1 = stream1.getTimestamp();

            while ((frame = filter.pull()) != null) {
                long rt = recorder.getTimestamp();
                long ft = frame.timestamp;
                if (frame0.type == Frame.Type.VIDEO && frame1.type == Frame.Type.VIDEO) {
                    System.out.println("t0="+t0+"  t1="+t1+"  filter="+ft+"  recorder="+rt + "  type0="+frame0.type+ "  type1="+frame1.type);

                    recorder.setTimestamp(t1);
                    recorder.record(frame, filter.getPixelFormat());
                }
                else if (frame0.type == Frame.Type.AUDIO && frame1.type == Frame.Type.AUDIO) {

                    recorder.record(frame, filter.getPixelFormat());
                    System.err.println("------t0="+t0+"  t1="+t1+"  filter="+ft+"  recorder="+rt + "  type0="+frame0.type+ "  type1="+frame1.type);
                }

            }
        }

Here is the log:

t0=61009000  t1=61010000  filter=60920000  recorder=60966667  type0=VIDEO  type1=VIDEO
------t0=61127000  t1=61114000  filter=66385850  recorder=61033333  type0=AUDIO  type1=AUDIO
------t0=61170000  t1=61157000  filter=66432290  recorder=61033333  type0=AUDIO  type1=AUDIO
t0=61076000  t1=61076000  filter=61000000  recorder=61033333  type0=VIDEO  type1=VIDEO
------t0=61191000  t1=61178000  filter=66455510  recorder=61100000  type0=AUDIO  type1=AUDIO
------t0=61234000  t1=61221000  filter=66501950  recorder=61100000  type0=AUDIO  type1=AUDIO
------t0=61298000  t1=61285000  filter=66571609  recorder=61100000  type0=AUDIO  type1=AUDIO
------t0=61340000  t1=61327000  filter=66618049  recorder=61100000  type0=AUDIO  type1=AUDIO
t0=61243000  t1=61243000  filter=61160000  recorder=61100000  type0=VIDEO  type1=VIDEO
------t0=61362000  t1=61349000  filter=66641269  recorder=61266667  type0=AUDIO  type1=AUDIO
------t0=61404000  t1=61391000  filter=66687709  recorder=61266667  type0=AUDIO  type1=AUDIO
------t0=61426000  t1=61413000  filter=66710929  recorder=61266667  type0=AUDIO  type1=AUDIO
------t0=61468000  t1=61455000  filter=66757369  recorder=61266667  type0=AUDIO  type1=AUDIO
t0=61376000  t1=61376000  filter=61280000  recorder=61266667  type0=VIDEO  type1=VIDEO
------t0=61490000  t1=61477000  filter=66780589  recorder=61400000  type0=AUDIO  type1=AUDIO
------t0=61532000  t1=61519000  filter=66827029  recorder=61400000  type0=AUDIO  type1=AUDIO
------t0=61575000  t1=61562000  filter=66873469  recorder=61400000  type0=AUDIO  type1=AUDIO
t0=61476000  t1=61476000  filter=61400000  recorder=61400000  type0=VIDEO  type1=VIDEO
------t0=61596000  t1=61583000  filter=66896689  recorder=61500000  type0=AUDIO  type1=AUDIO
------t0=61639000  t1=61626000  filter=66943129  recorder=61500000  type0=AUDIO  type1=AUDIO
t0=61543000  t1=61543000  filter=61440000  recorder=61500000  type0=VIDEO  type1=VIDEO
------t0=61660000  t1=61647000  filter=66966349  recorder=61566667  type0=AUDIO  type1=AUDIO
------t0=61703000  t1=61690000  filter=67012789  recorder=61566667  type0=AUDIO  type1=AUDIO
t0=61609000  t1=61610000  filter=61520000  recorder=61566667  type0=VIDEO  type1=VIDEO
------t0=61724000  t1=61711000  filter=67036009  recorder=61633333  type0=AUDIO  type1=AUDIO
------t0=61767000  t1=61754000  filter=67082448  recorder=61633333  type0=AUDIO  type1=AUDIO
------t0=61831000  t1=61818000  filter=67152108  recorder=61633333  type0=AUDIO  type1=AUDIO
------t0=61874000  t1=61861000  filter=67198548  recorder=61633333  type0=AUDIO  type1=AUDIO
t0=61776000  t1=61776000  filter=61680000  recorder=61633333  type0=VIDEO  type1=VIDEO
------t0=61895000  t1=61882000  filter=67221768  recorder=61800000  type0=AUDIO  type1=AUDIO
------t0=61938000  t1=61925000  filter=67268208  recorder=61800000  type0=AUDIO  type1=AUDIO
t0=61843000  t1=61843000  filter=61760000  recorder=61800000  type0=VIDEO  type1=VIDEO
------t0=61959000  t1=61946000  filter=67291428  recorder=61866667  type0=AUDIO  type1=AUDIO
------t0=62002000  t1=61989000  filter=67337868  recorder=61866667  type0=AUDIO  type1=AUDIO
------t0=62023000  t1=62010000  filter=67361088  recorder=61866667  type0=AUDIO  type1=AUDIO

[saudet]

We can't set the timestamps for audio, you'll need to make sure to send the right amount of frames.

[libenli]

I revised the audio sampling rate of the filter and the recorder. It works normally. The timestamps of all frames of the print log are close. Thank you very much！