search

tsduck / tsduck

MPEG Transport Stream Toolkit
https://tsduck.io
BSD 2-Clause "Simplified" License
840 stars 210 forks source link

SCTE-35 injection #51

Closed jacksonvm closed 6 years ago

jacksonvm commented 6 years ago

Hi Thierry,

TSduck is an excellent contribution to the open source community and we are very much eager in exploring the possibilities. Is there any way we can inject SCTE-35 data in to a live mpegts stream?

Also is there are an option to MUX DVB subtitles using TSDuck ?

Regards,

Jackson

lelegard commented 6 years ago

Hi @jacksonvm,

Correct me if I am wrong, to inject SCTE 35, you need to modify the PMT and inject Splice Information Tables (I don't want to call it SIT since DVB already defines a SIT, Selection Information Table).

There are several possible ways to inject tables:

For the transformation of the PMT, the plugin pmt already has all appropriate options to add SCTE 35-related information, mostly thanks to code contribution from @haraldjordan78. Alternatively, you may recreate a complete PMT from scratch (or starting from a captured one) in XML format and then inject or replace it in the stream using the plugin inject.

For the Splice Information Table it is a bit different. The sections change over time because of the cues and they must be synchronised with the video PTS. Again, @haraldjordan78 contributed to code in the plugin mux, adding options to help in the synchronisation of sections and video.

But here, you must create and packetise the sections separately. TSDuck is not yet able to create Splice Information Tables because there is no XML definition for it. But it is not very difficult to write code to create those tables. To packetise these sections, you may use tspacketize. Alternatively, it you can use C++, the TSDuck library provides all services to create sections, tables and packetise them.

Now, for DVB subtitles, it is completely different. Here, you probably need a real MUX. In the open source world, ffmpeg may do the job although I ran into synchronisation issues with ffmpeg creating TS files with H.264 video. Otherwise, several non-free software MUX already exist but you probably know them already.

As a final word, injecting SCTE 35 is clearly a professional application (who would like ads insertion as a hobbyist :)). Although TSDuck is robust software with more than 10 years of development behind, it has no industrial quality certification. TSDuck was primarily developed for my own needs, as a toolbox for digital TV engineers, for test, integration support, debugging, demo, PoC and other non-production activities. Using TSDuck in a production tool would be done as your own risks. I would be confident enough to use TSDuck in production to work on offline files. However, for a live real time task such as inserting SCTE 35, you would have to use TSDuck to relay the complete production TS with, in case of failure, a complete black-out. If you want to do this, that will be at your own risks.

lelegard commented 6 years ago

Hi @jacksonvm (with @haraldjordan78 and @alexander-nf possibly interested).

Starting with commit 29542143cce01e2977ac3460a3d4a4d7b8c60fc7, the definition of <splice_information_table> has been completed in XML format. Consequently, SCTE 35 Splice Information Tables can now be created in XML format and compiled or packetized using TSDuck.

I am considering some spliceinject plugin to inject them in a live TS. Several points should be considered.

What would you suggest? Did I miss another important point?

jacksonvm commented 6 years ago

Hi @lelegard ,

Glad to hear from you. I havent worked on SCTE35 directly, but in the past we have built injectors to send SCTE104 to the Encoders and then the 104's are further converted in 35's and injected into the TS by the Encoders.

There are majorly 2 commands used in Splicing, one is SPLICE immediate and the other is SPLICE Normal. Splice Immediate gets inserted on the nearest I frame available, where as Splice Normal works in reference to the TIME CODE supplied to the encoder. The encoder cues the SCTE 104 Splice Normal command received till the scheduled time has reached.

emcodemall commented 6 years ago

Dear Thierry,

Sounds great! I would have loved to implement it that way. Instead i created binary SCTE Messages as files using attached python script. This script basically Supports just exactly my usecase but not more. SCTE35encoder.zip

Here some thoughts of mine:

lelegard commented 6 years ago

Hi,

Thanks for your feedbacks. Here is what I propose:

Input sections are accepted from files and UDP. File specification wildcard, polling interval, UDP port are specified in the command line. Files and UDP datagrams can contain binary sections or XML data (easy to check, the first byte is < for XML and 0xFC for binary). No problem, all coding basic blocks are already here from other parts of TSDuck.

Sample basic options: -P spliceinject --file '/foo/bar/scte*.xml' --udp [localaddress:]port

For insertion, all splice_insert() commands with PTS values (ie. not immediate) are inserted several times, at a specified interval, starting from a negative offset from the specified PTS (or earliest PTS when in component mode).

Sample basic options: --start-offset 2000 --count 2 --interval 1000 meaning that a splice_insert() is injected for the first time 2000 ms before the PTS it contains, injected twice at an interval of 1000 ms.

By doing like this, I see no use for an XML injection command. Unless of course you want this command injected once and that command injected twice, etc. But let's keep it simple.

All other commands, immediate splice_insert() and all other commands, are inserted immediately and exactly once.

I am not sure to correctly understand the semantics of the time_signal() command. Does it provide a real-time information on the PTS clock, just like a PCR as a system clock? In that case, it requires an extremely precise synchronisation which cannot be achieved the way I describe. Or does it provide an association between a PTS value and an absolute UTC/TAI time using an associated time_descriptor()? In the latter case, there is no stringent timing constraint.

lelegard commented 6 years ago

Just an additional information: The injection profile I propose for splice_insert() commands is based on what I observed on a Russian real-life stream which was provided by @alexander-nf in issue #20. Let me know if this profile is standard or not (whatever standard still means nowadays).

emcodemall commented 6 years ago

I am confused about the time_signal now as well. What i was referring to is the time_signal as a component of the Segmentation_descriptor. There it just Points to the intended target pts time which can be Offset by pts_adjustment value. No clue at about how it works as a standalone message ;-)

emcodemall commented 6 years ago

Just to be clear, you plan to parse all the found XML templates or binary files, get out their target PTS and use this as a base for the Offsets provided by the user, correct? As far as i understand this would match @alexander-nf 's and my streams pretty well! I guess this covers lots of usecases already. The only big deal i see is that all the descriptors would need to be implemented in the XML deserialisation. That seems like a whole bunch of work, so far i have not found any "complete" SCTE 35 Encoder or Decoder...

EDIT: fyi, i thought about this command XML because in my case, i needed to start continuous Insertion of commands between the end of last break and start of next break. Those were repeated every 4 secs and carried "segmentation_type_id = 0" in the UPID. Only 8 seconds before the start of break, one trigger with segmentation_type_id = 36 (Distributor Placement Opportunity Start) was inserted.

In that case, the --start-Offset would need to be calculated on a per-break Basis up in front. But it would still be achievable...

lelegard commented 6 years ago

Hi,

All splice descriptors are now implemented in XML. So any form of splice information table can be ingested from XML. Note that this is part of the TSDuck's XML syntax, not the SCTE 35-defined XML syntax.

The pts_adjustment value is not relevant here. When generating and inserting splice information sections, its value is always zero. It is here to be updated by downstream equipment in case of time restamping. Quote from SCTE 35:

pts_adjustment_ – A 33 bit unsigned integer that appears in the clear and that shall_ be used by a splicing device as an offset to be added to the (sometimes) encrypted pts_time field(s) throughout this message to obtain the intended splice time(s). When this field has a zero value, then the ptstime field(s) shall___ be used without an offset. Normally, the creator of a cueing message will place a zero value into this field.

This adjustment value is the means by which an upstream device, which restamps pcr/pts/dts, may convey to the splicing device the means by which to convert the pts_time field of the message to a newly imposed time domain.

When a downstream equipment performs time restamping, it cannot update a splice information section in the general case because the various PTS fields are in the possibly encrypted part. The pts_adjustment field is specifically placed in the clear part so that any equipment can patch it.

That being said, I don't really understand the need for encryption here. Being a "security person" myself, I hardly understand the threat analysis model here.

lelegard commented 6 years ago

Here is an analysis of the splice information section from the Russian stream in issue #20. The structure is very simple and there is no segmentation_descriptor. Here is a pair of out/in cues:

* SCTE 35 Splice Information, TID 252 (0xFC), PID 1045 (0x0415)
  Short section, total size: 50 bytes
  - Section 0:
    Protocol version: 0x00 (0)
    Encryption: none
    PTS adjustment: 0x000000000 (0)
    CW index: 0xFF (255), tier: 0xFFF (4095)
    Command type: 0x05 (SpliceInsert), size: 20 bytes
    Splice event id: 0x00000017, cancel: 0
    Out of network: yes, program splice: yes, duration set: yes, immediate: no
    Time PTS: 2,058,331,506
    Duration PTS: 0x0007B98A0 (8100000), auto return: yes
    Unique program id: 0x0001 (1), avail: 0x08 (8), avails expected: 255
    - Descriptor 0: SCTE 35 Avail (0x00, 0), 8 bytes
      Identifier: 0x43554549 ("CUEI")
      Provider id: 0x00000008
    CRC32: 0x4ADEEB3E (OK)

* SCTE 35 Splice Information, TID 252 (0xFC), PID 1045 (0x0415)
  Short section, total size: 45 bytes
  - Section 0:
    Protocol version: 0x00 (0)
    Encryption: none
    PTS adjustment: 0x000000000 (0)
    CW index: 0xFF (255), tier: 0xFFF (4095)
    Command type: 0x05 (SpliceInsert), size: 15 bytes
    Splice event id: 0x00000018, cancel: 0
    Out of network: no, program splice: yes, duration set: no, immediate: no
    Time PTS: 2,066,431,506
    Unique program id: 0x0001 (1), avail: 0x08 (8), avails expected: 255
    - Descriptor 0: SCTE 35 Avail (0x00, 0), 8 bytes
      Identifier: 0x43554549 ("CUEI")
      Provider id: 0x00000008
    CRC32: 0xD497DA82 (OK)

For a more complex workflow, could you please provide a precise scenario of injection?

lelegard commented 6 years ago

Hello @jacksonvm and @haraldjordan78,

I have pushed in commit fe3fd0088af66603baa7a7e563357c392546abc9 a working version of the spliceinject plugin. See the documentation in the user's guide.

If you'd like to test it, please pull and rebuild. Your feedback would be appreciated.

zuzzurro commented 6 years ago

Would it be possible to have somewhere an example splice XML that can be used for learning and inspiration?

lelegard commented 6 years ago

In the streams repository, there is one real-life example with an extracted XML file. This file contains all splice info tables from the PID. Typically, you would send the <splice_information_table> structures one by one for injection in spliceinject'.

zuzzurro commented 6 years ago

Thanks. I have been playing with that but knowing that that XML is in the right format is really helpful.

lelegard commented 6 years ago

That XML file was extracted from the stream using -P tables --xml (see the shell scripts in the stream repository page). All XML outputs which are produced by TSDuck are also valid inputs.

We rarely do things from scratch. We elaborate, modify or enhance existing stuff. This is why the XML format was added in TSDuck : to extract, modify and re-inject.

zuzzurro commented 6 years ago

OK, I'm trying to send a single very simple immediate cue point:

<?xml version="1.0" encoding="UTF-8"?>
<tsduck>
  <splice_information_table protocol_version="0" pts_adjustment="0" tier="0x0FFF">
    <splice_insert splice_event_id="0x00000001" out_of_network="true" splice_immediate="true" unique_program_id="0x0001">
      <break_duration auto_return="false" duration="6,750,000"/>
    </splice_insert>
    <splice_avail_descriptor identifier="0x43554549" provider_avail_id="0x00000012"/>
  </splice_information_table>
</tsduck>

But it fails with the following error:

Why is that?

lelegard commented 6 years ago

OK, so you just found a bug...

The message tells you (in a rather obscure way) that a <break_duration> is not allowed when the event is immediate. This is incorrect. I just checked the SCTE 35 standard and the syntax of the splice_insert() command allows this.

The fix is trivial. I just fixed it in commit 08a0af6fa7def7ba5da9f86e5f54e86c0e9ada1f and added the non-regression test in test-027.

If you want to test the fix, you may pull the current state of the repository. See here for build instructions.

Sorry for the inconvenience.

zuzzurro commented 6 years ago

Pulling and building is easy, I already had done it. Testing that the fix works is more complicated. Right now I'm trying to feed one stream with injected SCTE-35 to my elemental, but nothing has shown up just yet. :-(

Thanks a lot for the quick fix, anyway, and I will keep trying.

lelegard commented 6 years ago

Try to analyse the output using TSDuck and check that the splice info table are here.

Also make sure that you have the right signalisation in the PMT (registration descriptor and stream type).

zuzzurro commented 6 years ago

Yeah, I was sidetracked today by the bug and ended up reading the code and concluding that the issue was with my using the immediate mode. So I put back the original cue that actually was accepted by TSDuck, but not by my encoder. I will have to dig more to find the real issue. Your suggestions are actually quite good and I will try them as soon as I have time. Thanks.

zuzzurro commented 6 years ago

I love this tool... but.. 1) I started by git and compiled my copy, but then I ended up using the fedora rpm for the convenience. However now that I'm trying to use git again I found out that the rpm version installs all the shared objects plugins in /usr/bin... That's really bad. I think they should go under /usr/libexec/tsduck. 2) I tried using LD_LIBRARY_PATH for testing but here's is what I get:

paolo@wi-006-lx 2771> LD_LIBRARY_PATH=../../libtsduck/release-x86_64:../../tsplugins/release-x86_64 ./tsp
tsp: file.so: cannot open shared object file: No such file or directory
tsp: file.so: cannot open shared object file: No such file or directory

It looks the executable looks for file.so but the plugins is really

../../tsplugins/release-x86_64/tsplugin_file.so

I keep trying.

Now the reason for loving this tool: I tried again adding -P tables at the end of my command line and it worked, it prints the splice (I still need to find out why the encoder does not process it, but that I will check later). However I realized that the splice appeared after a long while. After thinking about it I move the spliecinject module afer the pcrbitrate and regulate modules and voilà, splices appear in real-time.

I love this tool..

lelegard commented 6 years ago

Hi,

Yes, the placement of plugins is questionable. I don't want to justify the choice, but I can explain it. I needed to address all the following cases:

Each environment has its own specificities and usage rules. Each application with plugins has its own choices. I wanted something simple and reliable. It did not want a "configure"-based project with a pre-build script. So, the simplest choice was to search the plugins in the same directory as the current executable, whichever it is.

BTW, I am still waiting for a Mac user complaining that the plugin shared libraries end up in .so instead of .dylib like any other dynamic library on macOS...

Concerning running the application from the build environment, see here.

You may also install it anywhere without building the rpm or deb package (see here).

For the exact rules of plugin search, have a look at the user's guide, at the end of the reference section for tsp.

Finally, concerning your splice issue, the tuning options should be different for real-time streams and offline files. With a real-time stream, you need to reduce the latency, at the expense of a slightly increased CPU load. The tsp command and some plugins have buffer size or burst size parameters. If you reduce them, you reduce the latency. But make sure to understand what they do first. Issue #54 describes a similar issue. You solved it in another way.

And thank you for your love :)

zuzzurro commented 6 years ago

I really really really love this tool. I have my script working fine now. I can add cue points to a stream taken from a file and the receiving encoder acts accordingly. I am using the splice immediate mode with looping increasing duration for test purposes. Thanks a lot for your support.

zuzzurro commented 6 years ago

I am pushing a 22Mbps TS to a Tandberg/Ericsson RX8200. It works mostly fine if I do this:

multicat z.ts 172.31.247.166:5099

This is what I get on the Ericsson's web console:

Input Status LOCKED 188 22.406 Mbits/s
Video Status RUNNING 1920x1080 Interlaced 29.970Hz 4:2:0 16:9 20.465 Mbits/s
Audio 1 Status RUNNING AAC

If I use this:

tsp --verbose -I file -i z.ts -P regulate -O ip 172.31.247.166:5099 -p 128
* regulate: minimum packet burst duration is 10,000,000 nano-seconds
* file: input bitrate is 22,405,573 b/s
* regulate: regulated at bitrate 22,405,573 b/s

this is what I get instead:

Input Status LOCKED 188 2.076 Mbits/s
Video Status STOPPED
Audio 1 Status STOPPED ---

And if I remove the '-p 128', it does not work at all.

I can use multicat+tsp and somehow it works, but it would be simpler if I could rely on a single tool for doing this.

By the way. The z.ts file has been recorded using multicat, but it seems that tsp can understand it fine.

lelegard commented 6 years ago

Hi,

-O ip -p 128 is a bit aggressive. The resultant UDP datagrams are 24 kB long. The default, 7 TS packets per UDP datagram, is the standard value. It is the highest number of packets to get Ethernet frames shorter than 1500 bytes, the Ethernet MTU (at east without jumbo frames).

So, I would have understood the opposite, working with 7, not working with 128.

First track : network bursts on bottlenecks. Second track : TS / UDP encapsulation.

First track: Check the network topology, from the output Ethernet port of your computer to the Ethernet input of your MUX. Is there any bottleneck, meaning moving from a higher to a lower bandwidth adapter or network segment? Example: Computer output = 1 Gb/s, MUX input: 100 Mb/s.

In that case, unregulated UDP bursts cannot pass into the bottleneck and packets are lost. Solution: reduce burst size, increase number of (smaller) bursts. Try and possibly adjust tuning values :

tsp -v \
    -I file -i z.ts --max-input-packets 128 --max-flushed-packets 128 \
    -P pcrbitrate \
    -P regulate -p 7 \
    -O ip 172.31.247.166:5099 -p 7

Second track : The ip plugin does not add any encapsulation. TS packets are simply sent in UDP datagrams. It this OK with your MUX? Try a network capture using Wireshark with multicat and tsp. Compare the encapsulation.

zuzzurro commented 6 years ago

I have to give you multiple answers, hopefully we'll be able to make some progress:

  1. To make your example I had to move -max-input-packets 128 --max-flushed-packets 128 right after -v. Otherwise I would get an error. In the meantime I thought it could be something you had just added so I pulled the repo and recompiled, at least we are aligned..
  2. I tried various -p values in the output starting from 7. The receivers starts locking something at -p10, but it's very unstable.
  3. Bottleneck. I am running the multicat command on the same pc as tsp. If there was a bottleneck it should affect both in a similar way, no? (apart from the encapsulation see next.
  4. Encapsulation. Yes, I was seeing a difference in encapsulation in wireshark between multicat and tsp.
  5. With multicat it shows a simple list of UDP packets:

3513265 703.350399225 172.31.2.226 172.31.247.166 UDP 1370 57381 → 5099 Len=1328

With tsp it shows 4004089 1018.098386624 172.31.2.226 172.31.247.166 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=11840, ID=27af) [Reassembled in #4004097]

zuzzurro commented 6 years ago

Sorry, I did not answer your last question. Multicat has a -U switch -U Destination has no RTP header Adding or removing makes no difference in behavior. I will check if it makes a difference in wireshark.

zuzzurro commented 6 years ago

And yes, -U makes it similar to tsp: 39999 18.792447398 172.31.2.226 172.31.247.166 MPEG TS 1358 [MP2T fragment of a reassembled packet] Still the receiver works fine in both cases.

lelegard commented 6 years ago

To make your example I had to move -max-input-packets 128 --max-flushed-packets 128 right after -v.

Yes, sorry about the option order. It is something I typed without trying. They are options for the tsp command, not plugin options. But you fixed it yourself.

Bottleneck. I am running the multicat command on the same pc as tsp. If there was a bottleneck it should affect both in a similar way, no?

Not really. It depends on the bursts. As long as the end-to-end bandwidth is lower than the capacity of the slowest element, the traffic may pass. But is depends on the sending cadence.

Let's take an example. A very theoretical one but this is just an explanation. Your output interface is 1 Gb/s. So is the LAN. But the input interface of the receiver is 100 Mb/s and has negotiated this b/w with the switch. You send 100 Mb in a burst of packets. At 1 Gb/s, you send everything in 0.1 second. These 100 Mb arrive on the switch. But the receiver input (and switch output) is 100 Mb/s and it takes 1 second to send. If the internal buffer in the switch can hold 100 Mb, fine. The switch buffer is filled in 0.1 s and emptied in 1.0 s. But if the buffer in the switch is 50 Mb, the second half is lost.

On the other hand, if you send 1 Mb every 0.01 second or 10 Mb every 0.1 second, each burst can fill in the buffer in the switch and you do not lose anything.

Thus, you may have two applications sending the same bandwidth with different burst profiles and one application works while the other does not. 

With multicat it shows a simple list of UDP packets:
3513265 703.350399225 172.31.2.226 172.31.247.166 UDP 1370 57381 → 5099 Len=1328

1370 = 54 + (7 * 188). So multicat sends 7 TS packets per UDP datagram. This is standard.

With tsp it shows
4004089 1018.098386624 172.31.2.226 172.31.247.166 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=11840, ID=27af) [Reassembled in #4004097]

1514 is the max size of an Ethernet frame but you specified a -p value greater than 7. So, the whole datagram is larger that the Ethernet MTU and the datagram was fragmented at IP layer. On an IP standpoint, this is standard. But, on an application standpoint, it is less robust because losing one fragment causes the loss of the whole datagram. This is why IP-TV uses 7 TS packets per UDP packet as a standard.

Now, this is strange. Usually, -P pcrbitrate -P regulate -p 7 -O ip -p 7 is supposed to solve the burst issue. I would investigate two tracks:

lelegard commented 6 years ago

Another suggestion: If the stream contains video in the clear, replace the MUX address with a multicast address and use VLC on another computer to read the UDP stream. Does it play correctly?

zuzzurro commented 6 years ago

Replies: 1) as I wrote above, using multicat with -U and tsp makes the wireshark traces identical (at first sight) same 1358 length, same protocol. 2) Yes, I never had any doubt that tsp produces correct output, BUT Data: As I wrote above the funny thing is that if I use multicat to read the file and send it to a multicast address and then I read that with tsp and send it to the RX8200, it works quite well. This seems to point to some sort of issue in the file input plugin?

$ multicat -U z.ts 226.59.61.1:5000
debug: socket definition: connect:226.59.61.1:5000

$ tsp -v \
    -I ip 226.59.61.1:5000  \
    -P pcrbitrate \
    -P regulate \
    -O ip 172.31.247.166:5099
* regulate: minimum packet burst duration is 10,000,000 nano-seconds
* ip: joining multicast group 226.59.61.1:5000 from local address 172.31.2.226
* ip: unknown input bitrate
* regulate: unknown bitrate, cannot regulate.
* pcrbitrate: new bitrate from PCR analysis: 22,405,725 b/s
* regulate: regulated at bitrate 22,405,725 b/s
* regulate: regulated at bitrate 22,405,725 b/s
* pcrbitrate: new bitrate from PCR analysis: 22,405,787 b/s
* regulate: regulated at bitrate 22,405,787 b/s
* pcrbitrate: new bitrate from PCR analysis: 22,373,919 b/s
Input Status LOCKED 188 22.407 Mbits/s
Video Status RUNNING 1920x1080 Interlaced 29.970Hz 4:2:0 16:9 20.465 Mbits/s
Audio 1 Status RUNNING AAC
lelegard commented 6 years ago

The input plugin file is very basic and used for years in similar scenarios. If course, a rare and subtle error is still possible but I wouldn't bet on this.

Does this work with this ?

tsp -v --max-input-packets 16 --max-flushed-packets 16 \
    -I file -i z.ts \
    -P pcrbitrate \
    -P regulate -p 7 \
    -O ip 172.31.247.166:5099

Could you try the VLC test I suggested?

jacksonvm commented 6 years ago

Hi @zuzzurro,

I too have been facing this issue a lot, But in my case i have been using FFMPEG to encode mpegts and send multicast to Ericsson Decoder. With Harmonic Decoder's it locks immediately and the signal is stable too.

My observations and suggestions are.

1) Try using Harmonic PIRD Proview 7100 or 8100. 2) Try using an old version FW on the Ericsson PIRD < 5.x ( i was able to lock on the older FW's)

@lelegard @zuzzurro I want to testout TSP by using FFMPEG as an ENCODER and sourcing IP INPUT to TSP and further add the SCTE35 on the fly on TSP and use the MPEGTS output to further package it to HLS with SCTE 35 markers. Will update you once i have set things up.

Keep up the good work.

Jackson

lelegard commented 6 years ago

TSP by using FFMPEG as an ENCODER and sourcing IP INPUT to TSP

I am not sure to understand what you are going to do. But if you want to inject FFMPEG output into TSP, a simple pipe is probably more reliable than IP. But maybe I completely misinterpret your intentions.

jacksonvm commented 6 years ago

I want to use FFmpeg as an encoder and use TSP to MUX SCTE35 realtime on a mpeg transport stream. I'll try piping the FFmpeg output to TSP.

Thanks,

Jackson

zuzzurro commented 6 years ago

Hi I'm still testing the workflow but I had to deal with other problems. In the meantime I have one question. I use multicat to receive a unicast stream and retransmit it in multicast Like this:

$ multicat -u -U @172.31.2.226:5001 225.0.0.91:5001      
debug: socket definition: bind:172.31.2.226:5001
debug: socket definition: connect:225.0.0.91:5001

and then I use another multicat to send it to another multicast address:

$ multicat -u -U @225.0.0.91:5001 225.0.0.91:5002      
debug: socket definition: bind:225.0.0.91:5001
debug: socket definition: connect:225.0.0.91:5002
^Cerror: poll error (Interrupted system call)

it works, but if I use tsp instead:

$ tsp -I ip 225.0.0.91:5001 -P regulate -p 32 -b 22000000 -O ip 225.0.0.91:5002              
* Error: ip: error binding socket to local address: Address already in use

it doesn't. That's a bit unexpected, is it? Strace for multicat shows:

bind(3, {sa_family=AF_INET, sin_port=htons(5001), sin_addr=inet_addr("225.0.0.91")}, 16) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [524288], 4) = 0
setsockopt(3, SOL_IP, IP_ADD_MEMBERSHIP, {imr_multiaddr=inet_addr("225.0.0.91"), imr_interface=inet_addr("0.0.0.0")}, 8) = 0
write(2, "debug: socket definition: connec"..., 50) = 50
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(4, SOL_SOCKET, SO_RCVBUF, [524288], 4) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(5002), sin_addr=inet_addr("225.0.0.91")}, 16) = 0

strace for tsp:

socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP) = 4
setsockopt(4, SOL_IP, IP_PKTINFO, [1], 4) = 0
setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
bind(4, {sa_family=AF_INET, sin_port=htons(5001), sin_addr=inet_addr("225.0.0.91")}, 16) = -1 EADDRINUSE (Address already in use)

From a superficial look the difference seems to be that multicat uses connect, while tsp uses bind..

lelegard commented 6 years ago

You need option --reuse-port.

zuzzurro commented 6 years ago

Damn.. I thought it would be easy

zuzzurro commented 6 years ago

OK, one more thing. If I connect using VLC to the stream generated by tsp after adding the --reuse-port option, it works, but the program in the Playback->Program menu are different (plain numbers vs channel names). I would have thought that tsp as I launched it would act as passthru? What other option I'm missing?

zuzzurro commented 6 years ago

Using:

tsp --max-input-packets 16 --max-flushed-packets 16 -I ip -r 225.0.0.91:5001 -P pcrbitrate -P regulate -p 7 -O ip 225.0.0.91:5002

for streaming to vlc works fine. (I'm using multicast IP since I source if live from another RX8200 and I trust this more than a recorded file).

Now I will try adding scte-35 and feed it to the receiving RX8200 to get SDI with scte-35 cues.

Thanks

xoraes commented 6 years ago

I am new and trying out tsduck to remove and inject scte-35 to a ts file. When I inspect removed-hello.ts it still has the splice information. What am I doing wrong?

`tsp -v -I file ~/tnt-2.ts -P rmsplice -a -s -O file removed-hello.ts

lelegard commented 6 years ago

@zuzzurro

If I connect using VLC to the stream generated by tsp after adding the --reuse-port option, it works, but the program in the Playback->Program menu are different (plain numbers vs channel names). I would have thought that tsp as I launched it would act as passthru?

Yes, without specific intermediate plugin which modifies packets, tsp is a complete passthru. Is it exactly the same stream in the two cases ? Is there any reason why the SDT could be removed ? If VLC can read the service names from the SDT from the first stream, why would it miss it in the second one ?

I suggest to also analyze the stream in parallel using tsp, in the two cases, something like:

tsp -I ip ... -P until --seconds 20 -P analyze -o analyze.txt -P psi -a -o psi.txt -O drop

Compare the two analyses and check that the service names are present in the analysis (analyze.txt) and in the SDT (psi.txt).

And BTW I should probably activate --reuse-port by default everywhere. It is almost always useful with multicast. Given your strace report, multicat activates it by default.

lelegard commented 6 years ago

@xoraes

There is a misunderstanding here. The plugin rmsplice does not remove the splice information. It removes the associated service based on splice information. More precisely, for a given service, when a "splice out" event occurs, all service PID's are dropped. They are passed again at the next "splice in" event.

The obvious use case is to remove ads from a recorded file.

I admit that this is mostly experimental since a broadcaster will likely remove the splice information after splice processing. So, if you record a TS at home, there will be no splice information and you will be obliged to remove ads manually using any video editor.

If you want to remove the splice information, you need to :

  1. Modify the PMT of the service to remove any reference to the splice info PID using -P pmt --remove-pid ...
  2. Drop the splice info PID using -P filter --negate --pid ...
xoraes commented 6 years ago

Thanks @lelegard . How about injection? I am a bit confused if is works since this bug is open but the documentation has spliceinject in it too. I am using the latest released version.

I have a question on spliceinject as well: I took a plain vanilla ts stream with no splice info and tried to do a spliceinject: tsp -I file ~/Desktop/hello.ts -P spliceinject --service 2245 -f inject*.xml -O file hello-inject.ts

When I verify hello-inject.ts for scte35 I don't see anything. Here is my inject file:

`

JUS5108-01

`

lelegard commented 6 years ago

Hi @xoraes

Your XML is not a valid TSDuck table. See the user's guide for the XML syntax of all tables and descriptors. So, if this file is actually ingested by spliceinsert, you should get an error message.

See appendix A.2.12 here and an real-life example here.

lelegard commented 6 years ago

@zuzzurro

I just modified the behaviour of option --reuse-port. It is now on by default everywhere, commands and plugins. A new option --no-reuse-port has been added, just in case you really need to disable it.

xoraes commented 6 years ago

@lelegard thanks for your help. I might still be doing something wrong because I added the xml from above link but no luck with the injection. Here is my truncated xml.

`<?xml version="1.0" encoding="UTF-8"?>

` Also, the tsp command does not show any error even if I have a completely wrong structure in the xml file.
zuzzurro commented 6 years ago

The new default makes a lot of sense in my opinion, in particular because tsp only supports multicast and with multicast my assumption is to be able to connect to the same stream multiple times. Only remaining small thing is why you don't support unicast at all..

lelegard commented 6 years ago

@zuzzurro

Only remaining small thing is why you don't support unicast at all.

TSDuck does support unicast. The syntax is -I ip [address:]port. The address is optional. If specified, it must be a multicast address and the plugin joins the corresponding group. Otherwise, without address, the plugin simply listens on the specified port and receives unicast traffic which is sent to the host on which tsp runs.

zuzzurro commented 6 years ago

Oh my god, sorry for spreading fake news one more time. I was misled by the error message:

$ tsp -I ip 172.31.2.226:5001 -O ip 172.31.247.1:5099
* Error: ip: address 172.31.2.226:5001 is not multicast

I just assumed that tsp did not support multicast at all while it just does not "need" the IP part.. I guess my brain is just primed by other tools behavior..