Handle streams with corrupted pts or dts timestamps

dzianis-dashkevich commented 1 year ago

Description

Sometimes, we may encounter streams with corrupted PTS/DTS timestamps during discontinuities.

Let's review the following tsanalyze dump based on these corrupted segments (This specific segment is the very first segment after discontinuity):

===============================================================================
|  TRANSPORT STREAM ANALYSIS REPORT                                           |
|=============================================================================|
|  Transport Stream Id: .......... 1 (0x0001)  |  Services: .............. 1  |
|  Bytes: ............................ 75,764  |  PID's: Total: .......... 5  |
|  TS packets: .......................... 403  |         Clear: .......... 5  |
|     With invalid sync: .................. 0  |         Scrambled: ...... 0  |
|     With transport error: ............... 0  |         With PCR's: ..... 0  |
|     Suspect and ignored: ................ 0  |         Unreferenced: ... 0  |
|-----------------------------------------------------------------------------|
|  Transport stream bitrate, based on ....... 188 bytes/pkt    204 bytes/pkt  |
|  User-specified: ................................... None             None  |
|  Estimated based on PCR's: ...................... Unknown          Unknown  |
|  Selected reference bitrate: .................... Unknown             None  |
|-----------------------------------------------------------------------------|
|  Broadcast time: ................................................. Unknown  |
|-----------------------------------------------------------------------------|
|  Srv Id  Service Name                              Access          Bitrate  |
|  0x0001  (unknown) .................................... C          Unknown  |
|                                                                             |
|  Note 1: C=Clear, S=Scrambled                                               |
|  Note 2: Unless specified otherwise, bitrates are based on 188 bytes/pkt    |
===============================================================================

===============================================================================
|  SERVICES ANALYSIS REPORT                                                   |
|=============================================================================|
|  Global PID's                                                               |
|  TS packets: 1, PID's: 1 (clear: 1, scrambled: 0)                           |
|-----------------------------------------------------------------------------|
|     PID  Usage                                     Access          Bitrate  |
|   Total  Global PID's ................................. C          Unknown  |
|   Subt.  Global PSI/SI PID's (0x00-0x1F) .............. C          Unknown  |
|  0x0000  PAT .......................................... C          Unknown  |
|=============================================================================|
|  Service: 0x0001 (1), TS: 0x0001 (1), Original Netw: 0x0000 (0)             |
|  Service name: (unknown), provider: (unknown)                               |
|  Service type: 0x00 (Undefined)                                             |
|  TS packets: 402, PID's: 4 (clear: 4, scrambled: 0)                         |
|  PMT PID: 0x0100 (256), PCR PID: 0x0101 (257)                               |
|-----------------------------------------------------------------------------|
|     PID  Usage                                     Access          Bitrate  |
|   Total  Undefined .................................... C          Unknown  |
|  0x0100  PMT .......................................... C          Unknown  |
|  0x0101  MPEG-2 AAC Audio ............................. C          Unknown  |
|  0x0102  AVC video (416x234, main profile, level 1.3,   C          Unknown  |
|  0x0103  MetaData in PES packets ...................... C          Unknown  |
|          (C=Clear, S=Scrambled, +=Shared)                                   |
===============================================================================

===============================================================================
|  PIDS ANALYSIS REPORT                                                       |
|=============================================================================|
|  PID: 0x0000 (0)                                                       PAT  |
|-----------------------------------------------------------------------------|
|  Global PID                Transport:                Discontinuities:       |
|  Bitrate: ....... Unknown  Packets: ............. 1  Expected: ......... 0  |
|  Access: .......... Clear  Adapt.F.: ............ 0  Unexpect: ......... 0  |
|                            Duplicated: .......... 0  Sections:              |
|                                                      Unit start: ....... 1  |
|=============================================================================|
|  PID: 0x0100 (256)                                                     PMT  |
|  Service: 0x0001 (1) (unknown)                                              |
|-----------------------------------------------------------------------------|
|  Single Service PID        Transport:                Discontinuities:       |
|  Bitrate: ....... Unknown  Packets: ............. 1  Expected: ......... 0  |
|  Access: .......... Clear  Adapt.F.: ............ 0  Unexpect: ......... 0  |
|                            Duplicated: .......... 0  Sections:              |
|                                                      Unit start: ....... 1  |
|=============================================================================|
|  PID: 0x0101 (257)                                        MPEG-2 AAC Audio  |
|  PES stream id: 0xC0 (Audio 0)                                              |
|  Service: 0x0001 (1) (unknown)                                              |
|-----------------------------------------------------------------------------|
|  Single Service PID        Transport:                Discontinuities:       |
|  Bitrate: ....... Unknown  Packets: ............ 80  Expected: ......... 0  |
|  Access: .......... Clear  Adapt.F.: ........... 44  Unexpect: ......... 0  |
|                            Duplicated: .......... 0  PES:                   |
|                                                      Packets: ......... 48  |
|                                                      Inv.Start: ........ 0  |
|  Clock values range:                                                        |
|                            PTS: ................ 48                         |
|                            from ................. 0                         |
|                            to ............. 180,480                         |
|                            Leaps: ............... 0                         |
|=============================================================================|
|  PID: 0x0102 (258)                                               AVC video  |
|  PES stream id: 0xE0 (Video 0)                                              |
|  416x234, main profile, level 1.3, 4:2:0                                    |
|  Service: 0x0001 (1) (unknown)                                              |
|-----------------------------------------------------------------------------|
|  Single Service PID        Transport:                Discontinuities:       |
|  Bitrate: ....... Unknown  Packets: ........... 320  Expected: ......... 0  |
|  Access: .......... Clear  Adapt.F.: ........... 33  Unexpect: ......... 0  |
|                            Duplicated: .......... 0  PES:                   |
|                                                      Packets: ......... 31  |
|                                                      Inv.Start: ........ 0  |
|  Clock values range:                                                        |
|                            PTS: ................ 31  DTS: ............. 31  |
|                            from ................. 0  from .. 8,589,916,592  |
|                            to ............. 177,075  to .......... 165,075  |
|                            Leaps: ............... 0  Leaps: ............ 1  |
|=============================================================================|
|  PID: 0x0103 (259)                                 MetaData in PES packets  |
|  PES stream id: 0xBD (Private stream 1)                                     |
|  Service: 0x0001 (1) (unknown)                                              |
|-----------------------------------------------------------------------------|
|  Single Service PID        Transport:                Discontinuities:       |
|  Bitrate: ....... Unknown  Packets: ............. 1  Expected: ......... 0  |
|  Access: .......... Clear  Adapt.F.: ............ 0  Unexpect: ......... 0  |
|                            Duplicated: .......... 0  PES:                   |
|                                                      Packets: .......... 1  |
|                                                      Inv.Start: ........ 0  |
|  Clock values range:                                                        |
|                            PTS: ................. 1                         |
|                            from ................. 0                         |
|                            to ................... 0                         |
|                            Leaps: ............... 0                         |
===============================================================================

===============================================================================
|  TABLES & SECTIONS ANALYSIS REPORT                                          |
|=============================================================================|
|  PID: 0x0000 (0)                                                       PAT  |
|-----------------------------------------------------------------------------|
|  0x00 (0, PAT), TID ext: 0x0001 (1)                                         |
|      Repetition: ...... 0  pkt  Section cnt: ........ 1                     |
|      Min repet.: ...... 0  pkt  Table cnt: .......... 1                     |
|      Max repet.: ...... 0  pkt  Version: ............ 0                     |
|=============================================================================|
|  PID: 0x0100 (256)                                                     PMT  |
|  Service: 0x0001 (1) (unknown)                                              |
|-----------------------------------------------------------------------------|
|  0x02 (2, PMT), TID ext: 0x0001 (1)                                         |
|      Repetition: ...... 0  pkt  Section cnt: ........ 1                     |
|      Min repet.: ...... 0  pkt  Table cnt: .......... 1                     |
|      Max repet.: ...... 0  pkt  Version: ............ 0                     |
===============================================================================

As you can see, Audio packets have only PTS, and it is monotonically increasing from 0 to 180,480. Video packets have both PTS and DTS values, and DTS is not monotonically increasing: It goes from 8,589,916,592 to 165,075.

Let's take a look closer at video packets with timestamp info from this segment using tsdump:

1st video packet with PTS/DTS info:

PTS is 0x000000000, which is 0, and DTS is 0x1FFFFB9B0, which is 8589916592 (33-bit)

* Packet 2
  ---- TS Header ----
  PID: 258 (0x0102), header size: 96, sync: 0x47
  Error: 0, unit start: 1, priority: 0
  Scrambling: 0, continuity counter: 0
  Adaptation field: yes (92 bytes), payload: yes (92 bytes)
  Discontinuity: 0, random access: 0, ES priority: 0
  ---- PES Header ----
  Stream id: 0xE0 (Video 0)
  PES packet length: 0 (unbounded)
  DTS: 0x1FFFFB9B0, PTS: 0x000000000 (DTS-95,443,517 ms)
  ---- Full TS Packet Content ----
  47 41 02 30 5B 00 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
  FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
  00 00 01 E0 00 00 84 C0 0A 31 00 01 00 01 1F FF FF 73 61 00 00 00 01 09
  10 00 00 00 01 67 4D 40 0D EC A0 D0 FF C9 35 01 01 01 A5 00 00 03 00 01
  00 00 03 00 1E 8F 14 29 96 00 00 00 01 68 EA EF 20 00 00 01 06 05 FF FF
  EF DC 45 E9 BD E6 D9 48 B7 96 2C D8 20 D9 23 EE EF 78 32 36

2nd video packet with PTS/DTS info:

PTS is 0x000001770, which is 6000, and DTS is 0x1FFFFD120, which is 8589922592 (increased by 6000, still 33-bit)

* Packet 22
  ---- TS Header ----
  PID: 258 (0x0102), header size: 4, sync: 0x47
  Error: 0, unit start: 1, priority: 0
  Scrambling: 0, continuity counter: 4
  Adaptation field: no (0 bytes), payload: yes (184 bytes)
  ---- PES Header ----
  Stream id: 0xE0 (Video 0)
  PES packet length: 0 (unbounded)
  DTS: 0x1FFFFD120, PTS: 0x000001770 (DTS-95,443,517 ms)
  ---- Full TS Packet Content ----
  47 41 02 14 00 00 01 E0 00 00 84 C0 0A 31 00 01 2E E1 1F FF FF A2 41 00
  00 00 01 09 30 00 00 01 41 9A 21 6C 44 7F DD E6 67 C8 FB 94 B9 FF 3E 99
  5C BF C3 55 69 8C 41 CD 0B 0B E9 FC B8 33 65 99 29 12 83 31 6B BC 2B AD
  F5 C5 C3 A4 81 00 3C EE 2A E9 B2 3B A4 C3 88 C2 46 18 85 8E 94 78 69 63
  44 BC 24 98 47 15 A5 59 44 59 41 88 72 42 7C 61 26 E0 72 92 80 0A C2 49
  EE B7 D5 8F 7C 70 E4 E2 5B BE 36 84 9A DC CE DF 27 15 86 4B 53 32 5E 1E
  BA AC 29 4C E6 36 A2 CC 2D E5 F9 E6 C5 32 1D 10 19 36 D6 93 2F C9 31 DC
  4E 90 59 AA EA A6 5D 59 CC 63 93 0B 6F 6F 48 80 A3 80 8A 74

3rd video packet with PTS/DTS info:

PTS is 0x000002EE0, which is 12000, and DTS is 0x1FFFFE890, which is 8589928592 (increased by 6000, still 33-bit)

* Packet 29
  ---- TS Header ----
  PID: 258 (0x0102), header size: 4, sync: 0x47
  Error: 0, unit start: 1, priority: 0
  Scrambling: 0, continuity counter: 11
  Adaptation field: no (0 bytes), payload: yes (184 bytes)
  ---- PES Header ----
  Stream id: 0xE0 (Video 0)
  PES packet length: 0 (unbounded)
  DTS: 0x1FFFFE890, PTS: 0x000002EE0 (DTS-95,443,517 ms)
  ---- Full TS Packet Content ----
  47 41 02 1B 00 00 01 E0 00 00 84 C0 0A 31 00 01 5D C1 1F FF FF D1 21 00
  00 00 01 09 30 00 00 01 41 9A 42 18 26 FF D4 C3 C8 03 28 91 BC 9E 08 B8
  8D 78 81 A7 27 7C 46 D5 50 F2 06 AE DE DE 25 EB B9 AB 3C 4F F1 25 BE 1B
  1D 9C CB AC FE 50 84 DD F5 98 09 2E E3 9C ED DB 56 05 E3 90 27 20 35 0D
  A6 13 64 87 75 D0 A0 7D 96 8A 88 EC 26 A2 8C 82 BE 59 FD B9 3A 21 BC 57
  43 59 5E 05 ED DB 56 D8 68 E9 B2 1B 57 06 8D 9D B1 0F E8 E0 A6 44 6D 19
  9F C5 52 DE 18 88 D6 58 C8 17 38 97 AE 48 90 FA EF 95 9A D4 23 D7 0D C0
  BB 68 E9 2E 42 6D 0F 70 55 05 F8 18 9D 42 94 28 CA 15 E8 AA

4th video packet with PTS/DTS info:

PTS is 0x000004650, which is 18000, and DTS is 0x000000000, which is 0 (it seems like timestamp rollover because according to ISO/IEC 13818-1, DTS/PTS value should be a 33-bit number, but if we take previous value 8589928592 + 6000 changes it will be 8589934592, which is 1000000000000000000000000000000000 in binary and this is 34-bit value, so it is rolling over to 0)

* Packet 46
  ---- TS Header ----
  PID: 258 (0x0102), header size: 4, sync: 0x47
  Error: 0, unit start: 1, priority: 0
  Scrambling: 0, continuity counter: 11
  Adaptation field: no (0 bytes), payload: yes (184 bytes)
  ---- PES Header ----
  Stream id: 0xE0 (Video 0)
  PES packet length: 0 (unbounded)
  DTS: 0x000000000, PTS: 0x000004650 (DTS+200 ms)
  ---- Full TS Packet Content ----
  47 41 02 1B 00 00 01 E0 00 00 84 C0 0A 31 00 01 8C A1 11 00 01 00 01 00
  00 00 01 09 30 00 00 01 41 9A 63 18 26 FF D8 4E 46 D7 92 0A A9 04 4B 39
  9B 54 1B 16 1E 76 3C 78 32 81 EE 7B 0B 62 19 09 1F 37 91 E4 4C 2E 3B 19
  9C 09 8E 8A 43 4E 4B F8 53 02 6F 86 DC 17 C1 DF B9 49 47 23 8D 65 25 04
  6A D4 E6 80 73 CC 07 88 71 E2 F6 D5 BE D2 32 8B 65 02 1A 53 E4 9D 9B 36
  5C BD 8E 28 84 76 9F 3B A8 B8 E5 78 EB FF 39 12 14 A4 DC 68 CC C6 E5 FA
  72 72 3C D8 8C 60 CB C0 05 AE AC A9 77 A1 78 3F 81 8A 98 D4 5E 13 20 76
  22 0B BF C3 F8 9A C8 0C AE C6 4E 7F 28 49 7F 55 C9 D0 1E A4

Since mux.js relies heavily on DTS values, we have two problems here: Problem 1: Differences between audio and video packets' reported start time in the same segment from probeTs:

0 vs 8589916592

Problem 2: Wrong timestampOffset for next segment after corrupted one.

Since this is a discontinuity segment, we re-calculate timestampOffsets based on the segment's timestamps and apply them to source buffers. The next segment will have the same timeline as the corrupted segment. So We do not re-calculate timestampOffset, but since the timestamp value was rolled over, we should apply a different timestampOffset for proper time alignment.

Specific Changes proposed

Use audio/video start time info from Transmuxer instead of probeTs because Transmuxer aligns time between audio and video in the timestampRollover stream.
Add a feature flag to calculate timestampOffset for each segment, regardless of its timeline, so timestampOffset should always be valid.

I added a feature flag to be safer, but overall, I consider this change risky, as it may break something.

Requirements Checklist

[x] Feature implemented / Bug fixed
[ ] If necessary, more likely in a feature request than a bug fix
- [x] Unit Tests updated or fixed
- [x] Docs/guides updated
- [ ] Example created (starter template on JSBin)
[ ] Reviewed by Two Core Contributors

codecov[bot] commented 1 year ago

Codecov Report

Merging #1426 (1d60c53) into main (04451d4) will increase coverage by 0.05%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1426      +/-   ##
==========================================
+ Coverage   85.75%   85.81%   +0.05%     
==========================================
  Files          42       42              
  Lines       10284    10298      +14     
  Branches     2374     2377       +3     
==========================================
+ Hits         8819     8837      +18     
+ Misses       1465     1461       -4

Files Changed	Coverage Δ
src/media-segment-request.js	`96.11% <ø> (+0.61%)`	:arrow_up:
src/playlist-controller.js	`95.36% <ø> (ø)`
src/segment-loader.js	`96.69% <100.00%> (+0.19%)`	:arrow_up:
src/source-updater.js	`94.24% <100.00%> (+0.05%)`	:arrow_up:
src/util/vjs-compat.js	`100.00% <100.00%> (ø)`
src/videojs-http-streaming.js	`91.02% <100.00%> (+0.01%)`	:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

amtins commented 1 year ago

@dzianis-dashkevich I tested this PR in relation to #1371 and #1049 and it seems that it can also solve these two issues, allowing https://github.com/videojs/mux.js/pull/430 and https://github.com/videojs/mux.js/pull/437 to be closed. It would be great to have 1 or 2 unit tests.

dzianis-dashkevich commented 1 year ago

@dzianis-dashkevich I tested this PR in relation to #1371 and #1049 and it seems that it can also solve these two issues, allowing videojs/mux.js#430 and videojs/mux.js#437 to be closed. It would be great to have 1 or 2 unit tests.

@amtins Thank you for your review! I added unit tests.

Regarding closing PR in mux.js... Let's test these changes against many different types of streams to be confident that everything works fine first

dzianis-dashkevich commented 1 year ago

This is great! + 1 once all the tests are passing.

It seems like everything is passing now

videojs / http-streaming