search

ossrs / oryx

Oryx(SRS Stack) is an all-in-one, out-of-the-box, and open-source video solution for creating online video services, including live streaming and WebRTC, on the cloud or through self-hosting.
https://ossrs.io/oryx
MIT License
563 stars 118 forks source link

Transcript: Noisy introduced in audio stream. #222

Open winlinvip opened 2 months ago

winlinvip commented 2 months ago

Origin stream is OK:

original.zip

Transcript stream has noisy:

transcript.zip

The noisy comes from the transcoding for each piece, so it appears every 10s, when switching to next ts piece.

You can use VLC to play the index.m3u8 and reproduce this issue.

winlinvip commented 2 weeks ago

If change the fps to 25, the stutter is less but still exists.

suzp1984 commented 2 weeks ago

Play transcript.zip by ffplay: ffplay -loglevel debug index.m3u8, the warning message is

[hls @ 0x144e076c0] HLS request for url '1455-overlay-b5c12e9c-53d9-4cac-ac4f-04f08a6c8637.ts', offset 0, playlist 0 [hls @ 0x144e076c0] Opening '1455-overlay-b5c12e9c-53d9-4cac-ac4f-04f08a6c8637.ts' for reading [mpegts @ 0x144e07c40] Continuity check failed for pid 17 expected 4 got 0 [mpegts @ 0x144e07c40] Continuity check failed for pid 0 expected 4 got 0 [mpegts @ 0x144e07c40] Continuity check failed for pid 4096 expected 4 got 0 [mpegts @ 0x144e07c40] Continuity check failed for pid 256 expected 11 got 0 [mpegts @ 0x144e07c40] Packet corrupt (stream = 0, dts = 1310932497). [hls @ 0x144e076c0] Packet corrupt (stream = 0, dts = 1310929494).

debug the ffplay by gdb, the Packet corrupt log print in

https://github.com/FFmpeg/FFmpeg/blob/1d55f548466516a6a4ed4cc67e118e1c29c6ae5a/libavformat/demux.c#L577-L581C78

    if (pkt->flags & AV_PKT_FLAG_CORRUPT) {
        av_log(s, AV_LOG_WARNING,
               "Packet corrupt (stream = %d, dts = %s)%s.\n",
               pkt->stream_index, av_ts2str(pkt->dts),
               s->flags & AVFMT_FLAG_DISCARD_CORRUPT ? ", dropping it" : "");

the AV_PKT_FLAG_CORRUPT is set at: https://github.com/FFmpeg/FFmpeg/blob/1d55f548466516a6a4ed4cc67e118e1c29c6ae5a/libavformat/mpegts.c#L2814-L2822 when demux failed the continuity check.

[mpegts @ 0x144e07c40] Continuity check failed for pid 17 expected 4 got 0 [mpegts @ 0x144e07c40] Continuity check failed for pid 0 expected 4 got 0 [mpegts @ 0x144e07c40] Continuity check failed for pid 4096 expected 4 got 0 [mpegts @ 0x144e07c40] Continuity check failed for pid 256 expected 11 got 0

https://github.com/ossrs/srs/blob/7951bf3bd63a33ec5bfddfba39f9d6e2d476e381/trunk/src/kernel/srs_kernel_ts.hpp#L406-L420

There is a 4-bit size continuity_counter field in TS packet, in range [0, 15], to keep the order of ts packets per pid.

when ffmpeg transcode the ts segment file, the countinuity_counter recount from 0 per pid. And I didn't find the ffmpeg cmd or filter to config the start counter, [0, 15], per pid for mpeg-ts format yet. If countinuity_counter config is not the feature of ffmpeg then I need to mux the mpeg-ts packet by code.

  1. first, maintain a map[pid] = previous ts segmet's last countinuity_counter;
  2. add another queue to the oryx transcript task, which demux and mux the ts packets by recount countinuity_counter from map[pid] + 1, and rewrite map[pid] by current ts segment's last packet.

Confirmed both the countinuity_counter (cc) and timestamp impact this issue. Here is the cc fixed transcript sample:

transcript_cc_fixed.zip

The code to account cc manually: https://gist.github.com/suzp1984/37a356a3fec4e69531ec10fbcb4a627f

suzp1984 commented 2 weeks ago

compare transcript.zip and transcript_cc_fixed.zip in VLC.

The sound loss of "transcript.zip" is very obvious, while the sound of "transcript_cc_fixed.zip" is discontinuous between ts segments. The transcript_cc_fixed.zip only let cc count continually between ts, so I think if the timestamp between ts improved, I believe the sounds can play well.

the timestamp can be improved by choice 25 fps video source.

If change the fps to 25, the stutter is less but still exists.

Next action:

suzp1984 commented 2 weeks ago

Another testing I did to verify the sound discontinuous problem, check above comment, the cc count continually between ts can fix the obvious problem, but still sound still not perfect.

I use the original.zip to reproduce the same audio stream problem.

here is the steps to reproduce:

  1. transcode the ts segments in original.zip ffmpeg -i 1457-org-cfe71a34-a90d-4e3f-9ec8-b008223cc957.ts -c:v libx264 -profile:v main -preset:v medium -tune zerolatency -bf 0 -c:a copy -copyts -y aac_copy/1457-org-cfe71a34-a90d-4e3f-9ec8-b008223cc957.ts transcode the ts segment with similar method used in oryx, but the audio data is copy without transcode, the video data is transcoded without rendering srt file. apply above cmd to every ts segments, and copy index.m3u8 to target folder aac_copy/ later.
  2. play aac_copy/index.m3u8 in VLC, the audio stream stutter is reproduced so close to transcript.zip

  3. apply cc count to ts segments, aac_copy/*.ts: cp aac_copy/1454-org-024c7fd8-9b60-431c-9ef6-236518f00078.ts aac_copy_cc_fixed/1454-org-024c7fd8-9b60-431c-9ef6-236518f00078.ts the first ts need not to fixed. run https://gist.github.com/suzp1984/37a356a3fec4e69531ec10fbcb4a627f to find the latest cc number for audio and video stream. go run . aac_copy_cc_fixed/1454-org-024c7fd8-9b60-431c-9ef6-236518f00078.ts change the video stream pid (256) and audio stream pid (257) new start count number according to above probe result. then run go run aac_copy/1455-*.ts aac_copy_cc_fixed/1455-*.ts redo similar probe for aac_copy_cc_fixed/1455-*.ts: go run . aac_copy_cc_fixed/1455-*.ts. then do same job to 1456-*.ts.

    1. play aac_copy_cc_fixed/index.m3u8 in VLC, the stream works really well.

    I know those steps are all did manually and so silly. I upload the result.

aac_copy.zip is the result of step 1, which reproduce the audio stream stutter. aac_copy_cc_fixed.zip is the result of step 3, which apply cc fix to step 1, the audio stream stutter fixed.

Conclusion The cc discontinuously between ts segments and audio stream transcode both contribute the audio stutter.