regulate plugin: missing option

[lars18th]

Hi @lelegard ,

I feel something needs to be improved in the pcrbitrate and regulate plugins. Let me to explain it:

From the 5.2.4 section of the TSDuck documentation:

tsp --max-input-packets 128 \
 –I dvb –c arte \
 -P zap arte \
 -P pcrbitrate --min-pcr 256 \
 -P regulate --packet-burst 128 \
 -O ip 224.10.11.12:1000 --packet-burst 128

We can assume that:

• The input (from a DVB capture device) is a MPTS with CBR.
• The output of the "arte" program is a SPTS with VBR.

Then I'm missing some options:

1. The pcrbitrate plugin recalculates the "bitrate" processing variable every 5 seconds by default (based on the missing global value of --bitrate-adjust-interval (page 47). However, to accomplish the task of accurate timing it's required to use the parameter -d to use the DTS clock, and recalculate the "bitrate" at every incoming packet with DTS marks. So, I feel the pcrbirate plugin needs to support the option --realtime to achieve this. Or if not, then the global value needs to be recalculated every 50 or 100ms. But the --bitrate-adjust-interval is expressed in seconds, not miliseconds.
2. The output bitstream from the previous example doesn't have correct PCR values. Some current plugins have support for recompute PCR marks (merge and rmsplice); but this isn't true for the regulate plugin. So in the previous example the output stream will not have correct PCR timestamps when the SPTS is VBR. In this case adding the option --adjust-time to the regulate plugin can solve the problem.

Please, comment if I'm wrong or these changes are necessary. The question is: I really need to have precise PCR values to complete a correct MUXING? I feel the merge plugin is at time useless when using files or other inputs not comming from live signals because the lack of these two options.

Regards.

[lelegard]

Hi @lars18th,

The option --bitrate-adjust-interval is used only for the input plugin. Bitrates are propagated from plugin to plugin. Each time a plugin computes a new bitrate value, it is immediately propagated downward; there is no bitrate adjustment interval between plugins. But most plugins do not recompute the bitrate and simply pass the previous value (except the plugin pcrbitrate, the sole purpose of which is precisely to recompute the bitrate). But this chain must be initialized somewhere. And the option --bitrate-adjust-interval gives the initial rythm.

But it does not apply here for two reasons.

1. The input plugin is a tuner and the input bitrate is fixed by definition. It never changes, whichever value --bitrate-adjust-interval has.

2. The plugin pcrbitrate completely ignores the bitrate value coming from the previous plugin in the chain, and the rythm of its variations. The plugin pcrbitrate assumes that the previous bitrate is incorrect and recomputes it. The plugin pcrbitrate reports a new bitrate value each time the computed value (based on PCR's) is "sufficiently" different from the previous value. Here, the threshold is hard-coded to 1 / 500 000. On a 50 Mb/s bitrate, the plugin pcrbitrate consequently reports a new value when this new value differs by 100 b/s.

Finally, I am not sure that the output PCR's are "incorrect". What does "incorrect" mean for a PCR? Doesn't it depend on the nature of a stream? Shall the PCR's be adapted to the stream? Or shall the stream be adapted to the PCR's?

For a CRB stream, like the one which goes into a modulator, the PCR's must be recomputed to be accurate according to the TS because the TS is synchronous and defines the clock.

But what about a VBR stream going out through IP? The stream does not define the clock because it is VBR. On the contrary, the PCR's inside the stream define the clock and the output of the stream shall be adapted to this clock.

So, my opinion is that, in the example, the PCR's should not be changed. On the contrary, the PCR's should be used to synchronize, accelerate or slow down, the output. And this is precisely what -P pcrbitrate -P regulate is supposed to do. Now, does it do it properly and accurately? This is another story.

But this is only my opinion. I may be wrong and I am ready to accept a demonstration proving that I am wrong.

[lars18th]

Hi @lelegard ,

Thank you for your explanation. For sure, I'm not thinking that you're wrong. Quite the opposite! I'm learning a lot from TSDuck and your help. :smile:

However let me to explain more the user case:

• The objective is do a MPTS muxing. In this case, all the inputs need to have accurate timings. Right?
• Today is very common that services are VBR, so the inputs before the muxing (merge plugin) require proper timing marks. True?
• Inputs can have diffent sources: files, RF demodulators, network multicasts, etc. Now, analyze the case of an HLS source. You use some HTTP client that translates from HLS to continous TS streaming. This HTTP output flow can now be injected in the tsp chain. But, because the nature of MPTS you need to feed this stream with a very accurate (around 50ms) bitrate. If not, then the decoder can suffer buffer problems.

So, this is in fact, the jitter problem with TS streams. No? Which tool can we use tp solve this problem? My current idea is to use pcrbitrate -d to adjust the streaming with "rendering" clocks. I feel this works. However, if I use an external tool for muxing, in this case the PCR of the output stream (after the tsp processing) needs to be adapted to it. And now I can't found any plugin to achieve this task. This is the reason to request for -P regulate --adjust-time.

This has sense? :confused:

[lelegard]

OK, I think I get it this time (better later than never...)

What we need in this case is synchronizing the transmission speed to the TS system clock (PCR). Using some option, regulate could wait on PCR values instead of average bitrate values. For instance:

• New option --pcr-synchronous to use PCR synchronization instead of bitrate.
• Select a reference PID for the clock. First PID containing some PCR's by default or option --pid-pcr.
• Each time a packet contains a PCR value from the reference PID, make sure it is not transmitted before its clock time.

Here, synchronizing on PCR's seems better than using DTS.

Potential issues:

• Precision will probably be no better than 20 ms in practice, due to operating system constraints.
• A "standard" Linux kernel cannot be more precise than 10 ms, plus timer latency.
• The default precision of a Windows kernel is 20 ms but it is not hard-coded and can be modified by the application. We can expect 1 or 2 ms I think.
• Do we need to smoothen packets between PCR's? Based on which information? Bitrate? But because of addition of timer latencies, this could be worse than doing nothing.

An using pcrbitrate would be superfluous now since its sole purpose is to generate a bitrate information for later use. And now, regulate would use absolute PCR values, not bitrate averages.

[lars18th]

Hi @lelegard ,

Yes, you got it this time! :+1: Futhermore, you explain it more precisely... the key are the PCR's and not the bitrate. :open_mouth:

And yes, I agree with the use of --pcr-synchronous and --pid-pcr options. They are perfect. However, please consider to add the option --rewrite-pcr too. I'm not sure if this task needs to be done in this plugin or in the output. But in any case, we need to have precise PCR marks in the output stream. Right?

Potential issues:

Let me to comment my opinion about them:

Precision will probably be no better than 20 ms in practice, due to operating system constraints.

As you're using an input buffer, you only need to measure PCR marks between two consecutive values with the output bitrate of the stream. If the bitrate is adjusted by the plugin, then you really know the value (total number of output packets) between the two timing marks. So the algoritm is pure deterministic and not based on time (PCR-1, PCR-2 and OUT-BYTES in the interval). The key is to have sufficient PCR's (3 or 4) waiting in the buffer.

A "standard" Linux kernel cannot be more precise than 10 ms, plus timer latency. The default precision of a Windows kernel is 20 ms but it is not hard-coded and can be modified by the application. We can expect 1 or 2 ms I think.

For this reason I recommend to process in chunks of 100ms (or 50ms)... or if you prefer, let the user to select the interval with one parameter.

Do we need to smoothen packets between PCR's? Based on which information? Bitrate? But because of addition of timer latencies, this could be worse than doing nothing.

For simplicity I suggest to not smooth between PCR's. This is a complex task. And a MUXER has the responsability of doing it. However, the idea here is to no go more fast than PCR's. So you can pass as fast (or slow) as you like between the two PCR's.

An using pcrbitrate would be superfluous now since its sole purpose is to generate a bitrate information for later use. And now, regulate would use absolute PCR values, not bitrate averages.

This is that I'm seeing. The pcrbitrate has no sense in a muxer chain.

[lelegard]

please consider to add the option --rewrite-pcr too.

There is a "chicken and egg" issue here. Bitrate and PCR's are 2 dependent variables. So, you need to choose which one you trust in order to modify the other :

• Either you trust the PCR values and you synchronize the transmission speed on them. Which is what you requested (or at least this is what I proposed and you agreed on it).
• Or you trust the bitrate information to be accurate and you can modify the PCR's based on this bitrate.

The scenario / spec I proposed was entirely dependent on the fact that you assume that the PCR are correct. So, I cannot rewrite them since they are correct by hypothesis.

[lars18th]

Hi @lelegard ,

I'm very busy... I'll comment other issues asap. But regarding this one...

There is a "chicken and egg" issue here. Bitrate and PCR's are 2 dependent variables. So, you need to choose which one you trust in order to modify the other :

The --rewrite-pcr option has sense in this scenario: when you are using CBR output and you need to recompute the PCR's values. You really like to do this after a muxing (aka merge) and before send it to a modulator. In fact, I feel some hardware modulators do this task inside the internal driver. So, for a dumb (or transparent) modulator this task is required.

So, to not mix concepts:

• PCR rewrite with --rewrite-pcr is for the final steps, and you trust the output bitrate.
• The PCR reading with --pcr-synchronous is for smoothing the stream at the initial steps of the chain; tipically before the muxing.
• --rewrite-pcr targets a pid with PCR's marks (so the TS can be SPTS or MPTS).
• --pcr-synchronous targets the entire stream and uses the pid identified with --pid-pcr (so it has sense with SPTS CBR or VBR, and MPTS only with CBR).

I hope it's more clear now.

[lelegard]

So, we agree that --pcr-synchronous is an option for the plugin regulate? This is just another time reference to use during the wait.

And --rewrite-pcr has nothing to do with wait. So, this is for another plugin. And this would be accurate only when the TS has CBR.

Typically, -P regulate --pcr-synchronous would be used in a secondary stream, the kind of command you use in a merge plugin.

And PCR rewrite would be used on the final stream after merge. But note that this PCR restamping is already done in merge. The only issue is that the bitrate information may not be really accurate, which is the topic of issue #76.

[lars18th]

Hi @lelegard ,

You're 100% right! :+1:

So --pcr-synchronous is for -P regulate (with the help of --pid-pcr). I feel this is the final conclusion of this issue. All other comments here should apply to #76 .

Now, as it's clear I hope you can implement it. :smile:

[lelegard]

Done in commit 6e9149dc26a84934d5da575c9582173d8fa38b86

I close this one. Please open a new issue in case of a specific problem with the new regulate options.