"PMT x does not exist" and "Adapter mismatch 4 != 0" after switching channels

[Jalle19]

I have a CAM that can decrypt 4 channels simultaneously. I run minisatip with -c 4:2 to accomplish this (two channels per PMT, the way I understand it).

Everything works fine most of the time, but sometimes, when I start a second or third channel simultaneously, minisatip gets so confused that all decryption stops working and the only remedy is to restart it.

When the bug occurs, these two lines start occuring:

[27/02 21:42:09.230 AD4]: PMT 4 does not exist
[27/02 21:42:09.230 AD4]: PMT 14 does not exist
...
[27/02 21:42:09.404 AD4]: Adapter mismatch 4 != 0
[27/02 21:42:09.404 AD4]: PMT 14 does not exist
...
[27/02 21:42:38.326 AD4]: PMT 4 does not exist
[27/02 21:42:38.326 AD4]: Adapter mismatch 4 != 1

(PMT 14 has master PMT 4)

I have not been able to reproduce this reliably yet, will need more time to figure out the exact steps required. Any help on this would be greatly appreciated.

Complete log (issue starts happening at 21:42:07): minisatip_adapter_mismatch.log

[Jalle19]

Forgot to attach the full log 🤦‍♂️ ~Will update the description.~ Edit: updated

[Jalle19]

I'm wondering if it's a race condition

[Jalle19]

I have a feeling that this happens when a new PAT is encountered (process_pat). The PAT doesn't contain the PMTs that are currently being descrambled by the CAM, so they get marked for deletion. Once they're deleted, the "PMT x does not exist" errors start showing up and the descrambling for the now deleted PMTs stop.

@catalinii any advice on how to fix this? I'm very slowly starting to grasp how the whole thing works but the interaction between pmt.c, ca.c and ddci.c is at times unclear.

[catalinii]

Hey, I need to look into this but been busy (moving) lately.

Thanks

[Jalle19]

No worries, I'm looking into it as well. Gonna try fiddling with logic that marks PMTs as "seen" and see if the problem goes away, then at least we'd know if I'm on the right track.

[Jalle19]

Good news! I commented out this whole block: https://github.com/catalinii/minisatip/blob/master/src/pmt.c#L1522 and it seems like descrambling never stops for existing streams no matter how many times I open or close different channels. There are other unwanted things in the log such as "DDCI 4 dropping packets for adapter X", and I have a feeling at some point some array will become full since we never delete PMTs, but it's a step in the right direction.

[catalinii]

The flow is something like this for 2 parallel tracks:

Common path: [DVB/SATIPC] (TS)-> original adapter buffer. First Path - related to PMT processing: process_pmt->start_pmt->send_pmt_to_cas->ddci_process_pmt Here basically a list of pmts are added to a DDCI device (ddci->pmt).

The other path is: Adapter buffer -> ddci_ts->(ts is rewritten to new pids to ensure multiple channels dont add same pids + pat/pmts generated based on ddci->pmt) -> DDCI hw device->ddci adapter buffer -> process_pat->process_pmt (for the ddci adapter).

ddci_adapter buffer->pid rewriting to original pids->original adapter buffer

That line that you mentions detects the new channel added/or removed to the ddci->pmt and potentially starts or updates a new pmt for that channel

[Jalle19]

Thanks for the explanation. I'm not exactly sure what the correct solution to this problem is though?

[catalinii]

I think a repro could help, but seems something may be off with the pid associated to the adapter for the pmt that the error is thrown.

[Jalle19]

I'm not sure what exact steps are required to reproduce this, but I think the key is enabling multiple PMTs for a CAM and then opening three or more channels simultaneously while cycling through them one by one (i.e. start three channels, close one, start another, close two, start two others etc.). At some point the "PMT x does not exist" errors start occuring and there is no recovery.

[catalinii]

We can do one thing. Generate the log and when minisatip is in a bad state we can generate a core dump of the process.

[Jalle19]

I'll see if I can find some time this week to do this. Bit hard to find "quiet" periods when no one's watching anything (and I don't have a dedicated "playground" setup either).

[catalinii]

If you could run it in a docker container and provide the Dockerfile would be great (to ensure everything is the same on both environments)

[Jalle19]

It's complicated since this involves a CI adapter. I only have one.

[Jalle19]

Trying to debug this again. I've found this strange piece of code:

    if (pmt->version == ver) {
        // just for testing purposes
        p = find_pid(pmt->adapter, pid);
        if (p)
            p->pmt = -pmt->id;

        if (ad && p && opts.clean_psi)
            clean_psi(ad, b, pid);

        return 0;
    }

If p->pmt is 14, the value gets updated to -14. This doesn't seem to make any sense? @catalinii do you know why this was added?

[Jalle19]

https://github.com/Jalle19/minisatip/commit/967df9718562b99b38f2f113cc982bde107fbcbe haven't tried this change yet, but it seems to be better than to always unconditionally screw with the value of p->pmt (where p is pointer to SPid)

[lars18th]

Hi @Jalle19 ,

This could be related (as a side effect) of bug #1043 ?

[Jalle19]

Seems to not be an issue anymore