SCTE-35 : timestamp adjustment is not used?

[cdongieux]

(Copy/paste from https://github.com/google/ExoPlayer/issues/11211 )

Hi,

I'm working on SCTE-35, it seems Metadata events (TimeSignalCommands) are not rendered at the right time, but earlier. I'm on ExoPlayer 2.18.6, I'm using DefaultRenderersFactory so MetadataRenderer.outputMetadataEarly is set to false.

I'm wondering if this is related to the fact that SpliceInfoDecoder does not use adjusted timestamps extracted from SpliceInsertCommand/TimeSignalCommand. Don't these adjusted timestamps should be used somewhere in the Metadata rendering chain for them to be fired at the right time?

Best regards

[cdongieux]

In deed, with something like this, Metadata events are rendered when expected:

In SpliceInfoDecoder:

  protected Metadata decode(MetadataInputBuffer inputBuffer, ByteBuffer buffer) {
    ....
    long presentationTimeUs = C.TIME_UNSET;
    switch (spliceCommandType) {
      case TYPE_SPLICE_NULL:
        command = new SpliceNullCommand(descriptors);
        break;
      case TYPE_SPLICE_SCHEDULE:
        command = SpliceScheduleCommand.parseFromSection(sectionData, descriptors);
        break;
      case TYPE_SPLICE_INSERT:
        command =
            SpliceInsertCommand.parseFromSection(sectionData, ptsAdjustment, timestampAdjuster,
                descriptors);
        break;
      case TYPE_TIME_SIGNAL:
        TimeSignalCommand timeSignalCommand = TimeSignalCommand.parseFromSection(sectionData, ptsAdjustment, timestampAdjuster,
            descriptors);
        presentationTimeUs = timeSignalCommand.playbackPositionUs;
        command = timeSignalCommand;
        break;
      case TYPE_PRIVATE_COMMAND:
        command = PrivateCommand.parseFromSection(sectionData, spliceCommandLength, ptsAdjustment,
            descriptors);
        break;
      default:
        // Do nothing.
        break;
    }
    return command == null ? new Metadata() : new Metadata(presentationTimeUs, command);
  }

And in MetadataRenderer:

  private void readMetadata() {
      ...
            if (!entries.isEmpty()) {
              long presentationTimeUs;
              if (metadata.presentationTimeUs == C.TIME_UNSET) {
                presentationTimeUs = getPresentationTimeUs(buffer.timeUs);
              } else {
                presentationTimeUs = metadata.presentationTimeUs;
              }
              Metadata expandedMetadata =
                  new Metadata(presentationTimeUs - 1000000000000L, entries);
              pendingMetadata = expandedMetadata;
            }
    ...
  }

[tianyif]

Hi @cdongieux, I'm going to close this issue since it is duplicated with https://github.com/google/ExoPlayer/issues/11211. Let's keep everything tracked in that one. Thanks!

[icbaker]

Let's keep this one and close the one in the exoplayer2 repo.

[icbaker]

I think this is working as intended. The metadata entries are emitted by MetadataRenderer at the time of their sample within the file, and the Entry contains the more precise timing info so an implementation of Player.Listener.onMetadata can choose to react at the 'correct' time (later) instead of when the callback is invoked. This gives implementations the 'forewarning' that is encoded in the SCTE-35 spec, specifically section 9.1 of SCTE 35 2016:

In order to give advance warning of the impending splice (a pre-roll function), the splice_insert() command could be sent multiple times before the splice point. For example, the splice_insert() command could be sent at 8, 5, 4 and 2 seconds prior to the packet containing the related splice point. In order to meet other splicing deadlines in the system, any message received with less than 4 seconds of advance notice may not create the desired result. The splice_insert() message shall be sent at least once a minimum of 4 seconds in advance of the desired splice time for a network Out Point condition. It is recommended that, if a return-to-network (an In Point) message is sent, the same minimum 4 second pre- roll be provided.

Since ExoPlayer doesn't directly implement any of the splicing reaction to SCTE-35 packets, it seems reasonable that an implementation of Player.Listener.onMetadata (that might be implementing this splicing) would need the same level of forewarning.

This is distinct from MetadataRenderer.outputMetadataEarly which outputs metadata as soon as it's available (i.e. when it's read from the input) rather than the current behaviour (which is to emit the metadata when the playback position reaches the 'sample' timestamp of the metadata entry).

It would be a potentially breaking changes for existing implementations of Player.Listener.onMetadata (which are reasonably expecting the packets to be emitted at their sample timestamp), to suddenly delay these SCTE-35 packets to their 'internal' timestamp.

[cdongieux]

OK, that makes sense.

But I'm wondering how to correctly handle timing info given in the Player.Listener.onMetadata callback in the TimeSignalCommand and SpliceInsertCommand cases. I'm seeing ptsTime and playbackPositionUs, playbackPositionUs seems to be the right value to deal with but it has a 1_000_000_000_000L offset. Is it related to MediaPeriodQueue.INITIAL_RENDERER_POSITION_OFFSET_US ? Shouldn't playbackPositionUs value be without this offset to be able to compare it to Player.getCurrentPosition()?

[icbaker]

Interesting question. SpliceInsertCommand.programSplicePlaybackPositionUs does seem to be in 'renderer time' ('playback time' isn't a very well defined term). Unfortunately making any fixed timestamp like this comparable to Player.getCurrentPosition() isn't really possible in the general case when considering live streams, because the 'origin' of getCurrentPosition() (i.e. zero point) moves with the live window of such a stream: https://developer.android.com/guide/topics/media/exoplayer/live-streaming

The current base of this timestamp is useful if you're in the Renderer domain, with access to Renderer.render(long, long), as this will be in the same base. Maybe if you're implementing splicing you will need to be making Renderer changes anyway? Not sure.

The other time base we could publish in would be 'period time', i.e. the time since the start of the current Period. This will be the same no matter where in a playlist the item is played (renderer time is always increasing across a playlist, so never consistent if the same item is played in different playlist positions). It also doesn't depend on live windows (unlike Player.getCurrentPosition()).

Are you able to flesh out a bit of context on what you're planning to do with the timestamp? That might help us decide what would make the most sense here, or help you to use the timestamps that are already available.

[cdongieux]

I'm experimenting things, but maybe that's not the better way. Let me recap.

My goal is to play replacement ads in place of ads from the original stream. Replacement ads are signaled by SCTE-35 events in the stream.

Some technical details. The input is a TS stream with a SCTE-35 PID. It is parsed by SpliceInfoDecoder (an enhanced version which implements unimplemented splice descriptors, I hope to make a PR soon). I have an instance of ExoPlayer which plays the TS stream and listens the Player.Listener.onMetadata SCTE-35 messages. As you said before, these messages are rendered by MetadataRenderer in advance of the moment they should be consumed, so I create PlayerMessages for them to be rendered at the time they should be consumed.

Code snippet:

val player: ExoPlayer = ...
player.addListener(object : Listener {
    override fun onMetadata(metadata: Metadata) {
        val position = player.currentPosition
        for (i in 0 until metadata.length()) {
            val entry = metadata[i]
            if (entry is SpliceCommand) {
                var forewarningDelayUs = 0L
                if (entry is TimeSignalCommand) {
                    forewarningDelayUs = entry.playbackPositionUs - 1_000_000_000_000L - metadata.presentationTimeUs
                    // identify splice descriptors and extract things I need
                    player.createMessage { messageType, _ ->
                        // notify something is happening
                        // like "entering an ads break"
                        // or "play the replacement ad number 3"
                        // etc.
                    }
                        .setType(Renderer.MSG_CUSTOM_BASE + 1)
                        .setPosition(position + forewarningDelayUs / 1000L)
                        .send()
                }
            }
    }
})

As you can see, I need to set the player position in created PlayerMessages. So in your view, what's the better way?