ffdecsawrapper in combination with mythtv 0.27 causes high CPU usage and failed recordings

[WiseguyTom]

I have stumbled on a problem which seems to be ffdecsawrapper related. I'm using mythtv 0.27 under Linux Mint 17 XFCE (kernel 3.13) and latest ffdecsawrapper git sources.

Symptoms: After some time (from a few minutes upto a few hours), mythtv and ffdecsawrapper start using a high CPU load.

• When watching LiveTV, the video freezes.
• When doing recordings on the backend (i.e. without running the frontend), they fail.

First, I tried all sorts of things to resolve the problem:

• Building and using the latest kernel from kernel.org (first applying the DVB patch of course)
• Using another PC (with the same DVB-S2 card)
• Using the latest mythtv, build from source
• Using the latest ffdecsawrapper, build from source They all show the same problem.

After spending days and late evenings debugging the mythtv and ffdecsawrapper code (I have some experience with kernel drivers and character device interfaces), I have discovered (and solved) a problem in ffdecsawrapper.

I discovered that the problem only occurs when there are transport errors in the MPEG transport stream (coming from the DVB-S2 card), i.e. when the Transport Error Indicator (TEI) is set in the MPEG transport stream header. See http://en.wikipedia.org/wiki/MPEG_transport_stream

So I did some monitoring on the transport stream. The TEI errors are only seen on one specific transponder, being Astra 19.2, 12721.75 (TV Vlaanderen). This is a rather busy transponder which is using a non standard symbol rate of 23500. I don't know if this has anything to do with it dough. At some times, about 0.1% of the packets contain a TEI error but during other parts of the day I do not see any errors at all, in which case the CPU load/freeze problem does not occur.

Anyhow, the ffdecsawrapper code should be able to cope with this but it doesn't :-)

When a packet contains such a TEI error, the PID (also a header field) is most often also invalid. I.e. the PID in the header isn't in use by any channel in the complete tuned transponder. Now there seems to be a flaw in the handling of unknown PID's in ffdecsawrapper.

The problem is located in process_ts() in dvblb_plugins/plugin_ffdecsa.c In short, an encrypted packet with an invalid PID can get 'stuck' in the ringbuffer when it's at the end of the buffer, finally overflowing the ringbuffer. I can explain the details later on ...

I've modified the code a bit and now the problem is gone :-) I would like to share this mod and contribute to the source code.

Cheers, Tom

[bas-t]

Hi Tom, If it does not break anything I'd be glad to make use of your patch. Please fork the repo, so you can do a pull request.

Tycho.

[bas-t]

Oh, and please do explain as much as you can!

[WiseguyTom]

I'll try to explain.

The decryption mechanism makes use of the ringbuffer implemented in plugin_ringbuf.c Note that a ringbuffer is not really circular but rather a linear buffer which wraps at the end. The ringbuffer has a read pointer (rdPtr) and a write pointer (wrPtr). It also has a begin pointer (buffer) and end pointer (end).

The decryption algorithm adds and extra pointer on top of that (csaPtr). In this discussion, the rdPtr is not relevant because it is simply lagging behind the csaPtr.

Now assume following condition (none means not encrypted):

Ringbuffer:
+----------------+  < buffer
| val pid, odd   | 0
+----------------+  < wrPtr
|   empty        | 1
+----------------+
~   empty        ~ ..
+----------------+  < csaPtr
| val pid, none  | 5318
+----------------+
| inv pid, odd   | 5319
+----------------+  < end

process_ffd() is called.

Because csaPtr > wrPtr (csaPtr is at the end of the ringbuffer and wrPtr has already wrapped), you do:

bytes = rb->end - csa->csaPtr;
end = 1;

And you call: process_ts(csa, buffer = csa->csaPtr, end = 2*188, force = 1)

Following piece of code skips the none encrypted packet:

while (pos < end)
{
  tmp=buffer[pos+3] & 0xC0;
  if(tmp == 0xc0 || tmp == 0x80)
    break;
  pos += TSPacketSIZE;
}

Then we enter the while loop for the invalid pid packet. ll_find_elem() returns NULL and the packet does not get assigned to an index, i.e. index stays at -1. We leave the while loop because no more packets are to be processed. We return 188 from process_ts(), the one none encrypted packet that we skipped. Note that nothing got decrypted because index stayed at -1 and rangeptr was not incremented. csaPtr is incremented with the one packet we skipped and process_ts() is called again with csaPtr pointing at the invalid pid packet. Same story, over and over now. So you call: process_ts(csa, buffer = csa->csaPtr, end = 1*188, force = 1) Nothing is skipped this time because the packet is encrypted. We enter the while loop (only one packet to be processed). Because it's and invalid pid, index end rangeptr do not get modified and nothing gets decrypted. We return zero from process_ts() So the invalid pid packet is stuck in the ringbuffer and never gets processed. I.e. csaPtr is no longer incremented.

All of this means that finally the ringbuffer will get full. The poll() system call gets into a loop because of this, hence the high CPU load in mythtv in the poll call().

The solution is to assign a default index to an invalid pid packet so that it always gets decrypted.

Regards, Tom

[WiseguyTom]

Note that I have cloned and updated the ffdecsawrapper repo under my account.

Regards, Tom

[bas-t]

Thanks for the patch and explanation. It' s in a test branch for a couple of days. When nothing strange happens, I' ll pull it to master.

Regards, Tycho