search

bas-t / descrambler

Standalone version of FFdecsawrapper
GNU General Public License v3.0
7 stars 11 forks source link

DMX and CA_PID improvements - no kernel header change required anymore #19

Open masjerang opened 3 years ago

masjerang commented 3 years ago

@Strunzdesign I've been playing a bit more with descrambler and did some changes to it (ie. remove DMX and ca_pid things depending on change to kernel headers). If you have some time, can you check and try: https://github.com/masjerang/descrambler Thanks!

Strunzdesign commented 3 years ago

Challenge accepted ;-)

Thank you very much, I just read the mail with the notification. I'm going to perform a quick review (hoping that I understand what you did, but I'm curious) then give it a try on my box. I may have a problem as my "kernel headers" of the system are "vanilla" as they are separate from the kernel source tree that I modified for the dvblookback driver. Is this an issue?

Regards, Florian

masjerang commented 3 years ago

Thanks! Good that we have a volunteer. :-) The idea is that you should be able to compile without modifying the vanilla kernel sources. Not sure if you easily ‘remove’ the patched kernel sources. This removes the need to patch sources outside descrambler. Please let me know your results.

Strunzdesign commented 3 years ago

Hi,

okay, now I'm confused. You write that I do not need to modify the vanilla kernel sources. What I currently do is this: I do need to patch them to add the dvbloopback driver, a process that involves the copying a driver directory, two "sed" commands, and the application of five patches. Omitting one of these steps breaks the compilation of the kernel and the contained dvbloopback driver.

Ok. Given that I managed to compile a kernel together with the dvbloopback module and successfully rebooted into the system. I end up with:

Reading your last comment, do you suggest to revert the patches of the kernel sources after the compilation (restoring the vanilla kernel sources + make clean + make distclean)? What do you mean that there is no need to patch sources "outside descrambler"? Outside descrambler, do you refer to the kernel containing the dvbloopback-related additions? Please clarify...

Regards, Florian

masjerang commented 3 years ago

Sorry, let me explain. Let split into dvbloopback and descrambler.

dvbloopback Normal build procedure, patching dvb-core and build dvbloopback, like you did before. No changes.

descrambler There was a patch called dmx.patch which patches the kernel sources (dmx.h and ca.h). Those patches were required to get descrambler compiling with a recent kernel. Instead of doing this dmx.patch, I've adapted descrambler to skip using the DMX stuff and added the required ca_pid structs etc to a separate header file calles ca_pid.h and included in descrambler header files. In order to compile the userspace binary descrambler, there is no need to patch any system source/header files anymore (for example, no root privileges required to build descrambler binary, but only need root to install).

Hope that makes more sense. Thanks!

Strunzdesign commented 3 years ago

Thank you very much for that clarification. Now I also see my mistake, that in fact I do patch the linux-headers package on my system, with the dmx-patch you mentioned. I forgot about it because some time ago I placed it into a "magic" folder on my Gentoo box to have it applied automatically by the package manager during each install / update of the linux-headers package. To summarize, I have to disable this dmx patch, reinstall linux-headers to revert the patch effectively, then give your repo a try. Straightforward, no problem.

I'm going to test that tomorrow as I my box is busy right now.

Regards, Florian

bas-t commented 3 years ago

@masjerang please do not do your development on the master branch. You should use a temporary branch to do that. Of course such temp branch should be initialy be a clone of the master branch. Should you need more info, let me know. Regards, Tycho.

masjerang commented 3 years ago

@bas-t and @Strunzdesign I've updated my fork and did how @bas-t recommened. My changes are now in testing-branch. Looking forward to testing.

warpme commented 3 years ago

Guys, I'm upgrading my server and decided to give a test for dvblopback+descrambler. OS is archlinux with 5.9.10 kernel. I compiled 5.9.10 kernel with dvblopback. it loads ok. Issue is with descrambler :-( Starting descrambler log:

Nov 26 12:34:33.952 INIT: Loading v1.0-10-gf30995a-Master
Nov 26 12:34:33.952 CAM: initializing FFdecsawrapper, A software emulated CAM
Nov 26 12:34:33.952 CAM(core.load): ** Key updates (AU) are enabled (active CAIDs) (no prestart)
Nov 26 12:34:33.952 CAM(core.load): ** Local systems DON'T take priority over cached remote
Nov 26 12:34:33.952 CAM(core.load): ** Force transfermode with digital audio
Nov 26 12:34:33.952 CAM(core.load): ** ECM cache is set to enabled
Nov 26 12:34:33.952 CAM(core.load): ** TsBufferSize is 32 MB
Nov 26 12:34:33.952 CAM(core.load): ** ScCaps are 1 2 0 0 0 0 0 0 0 0
Nov 26 12:34:33.952 CAM(general.info): loading overrides from /etc/sasc-ng/override.conf
Nov 26 12:34:33.952 CAM(core.load): loaded 0 overrides from /etc/sasc-ng/override.conf
Nov 26 12:34:33.952 CAM(general.info): loading smartcard data from /etc/sasc-ng/smartcard.conf
Nov 26 12:34:33.952 CAM(core.load): loaded 0 smartcard data from /etc/sasc-ng/smartcard.conf
Nov 26 12:34:33.952 CAM(general.info): loading cardslot config from /etc/sasc-ng/cardslot.conf
Nov 26 12:34:33.952 CAM(general.info): loading ecm cache from /etc/sasc-ng/ecm.cache
Nov 26 12:34:33.953 CAM(general.info): loading keys from /etc/sasc-ng/SoftCam.Key
Nov 26 12:34:33.953 CAM(core.load): loaded 0 keys from /etc/sasc-ng/SoftCam.Key
Nov 26 12:34:33.953 CAM(general.info): loading Viaccess cards from /etc/sasc-ng/Viaccess.KID
Nov 26 12:34:33.953 CAM(core.load): loaded 0 Viaccess cards from /etc/sasc-ng/Viaccess.KID
Nov 26 12:34:33.953 CAM(general.info): loading Seca cards from /etc/sasc-ng/Seca.KID
Nov 26 12:34:33.953 CAM(core.load): loaded 0 Seca cards from /etc/sasc-ng/Seca.KID
Nov 26 12:34:33.953 CAM(general.info): loading Irdeto cards from /etc/sasc-ng/Ird-Beta.KID
Nov 26 12:34:33.953 CAM(core.load): loaded 0 Irdeto cards from /etc/sasc-ng/Ird-Beta.KID
Nov 26 12:34:33.953 CAM(general.info): loading cardclient config from /etc/sasc-ng/cardclient.conf
Nov 26 12:34:33.953 CAM(cardclient.newcamd): now using protocol version 525 (cdLen=8)
Nov 26 12:34:33.953 CAM(cardclient.core): hostname=192.168.1.254 port=24000 emm=1 emmCaids 0100/ffff
Nov 26 12:34:33.953 CAM(cardclient.core): client 'Newcamd' ready
Nov 26 12:34:33.954 CAM(core.net): connecting to 192.168.1.254:24000/tcp (192.168.1.254)
Nov 26 12:34:33.954 THREAD: Netwatcher thread started (pid=574, tid=140013054121536)
Nov 26 12:34:33.957 CAM(cardclient.login): Newcamd: CaID=xxxx admin=1 srvUA=xxxxxxxxx provider 000000/0xxxxxxxxx
Nov 26 12:34:33.957 CAM(core.load): ** registered systems:
Nov 26 12:34:33.957 CAM(core.load): ** Viaccess          (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Seca              (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** SC-VideoGuard2    (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Viaccess       (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Seca           (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Nagra          (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Irdeto         (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Cryptoworks    (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** SC-Conax          (pri  -5)
Nov 26 12:34:33.957 CAM(core.load): ** Fake-NDS          (pri -12)
Nov 26 12:34:33.957 CAM(core.load): ** Nagra2            (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Nagra             (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Irdeto2           (pri  -8)
Nov 26 12:34:33.957 CAM(core.load): ** Irdeto            (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Cryptoworks       (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** ConstCW           (pri -20)
Nov 26 12:34:33.957 CAM(core.load): ** Conax             (pri -10)
Nov 26 12:34:33.957 CAM(core.load): ** Cardclient        (pri -15)
Nov 26 12:34:33.958 THREAD: SC housekeeper thread started (pid=574, tid=140013045728832)
Illegal instruction (core dumped)

Oops in kernel:

Nov 26 12:34:35 test systemd-coredump[578]: [LNK] Process 574 (ffdecsawrapper) of user 0 dumped core.

                                            Stack trace of thread 574:
                                            #0  0x000055e115bfe4bb _ZL18key_schedule_blockPhS_ (ffdecsawrapper + 0xca4bb)
                                            #1  0x000055e115bfed71 _ZL12schedule_keyP9csa_key_tPKh (ffdecsawrapper + 0xcad71)
                                            #2  0x000055e115c07411 _Z17set_control_wordsPvPKhS1_ (ffdecsawrapper + 0xd3411)
                                            #3  0x000055e115c0747a _Z14get_key_structv (ffdecsawrapper + 0xd347a)
                                            #4  0x000055e115beae90 connect_ffd (ffdecsawrapper + 0xb6e90)
                                            #5  0x000055e115b58e1e main (ffdecsawrapper + 0x24e1e)
                                            #6  0x00007f5755716152 __libc_start_main (libc.so.6 + 0x28152)
                                            #7  0x000055e115b5940e _start (ffdecsawrapper + 0x2540e)

                                            Stack trace of thread 575:
                                            #0  0x00007f57557b5c51 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xc7c51)
                                            #1  0x00007f57557bb137 __nanosleep (libc.so.6 + 0xcd137)
                                            #2  0x00007f57557bb06e sleep (libc.so.6 + 0xcd06e)
                                            #3  0x000055e115bd6bd8 _ZN11cNetWatcher6ActionEv (ffdecsawrapper + 0xa2bd8)
                                            #4  0x000055e115beee75 _ZN7cThread11StartThreadEPS_ (ffdecsawrapper + 0xbae75)
                                            #5  0x00007f5755bff3e9 start_thread (libpthread.so.0 + 0x93e9)
                                            #6  0x00007f57557ee293 __clone (libc.so.6 + 0x100293)

                                            Stack trace of thread 576:
                                            #0  0x00007f5755c059c8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf9c8)
                                            #1  0x000055e115bef00f _ZN9cCondWait4WaitEi (ffdecsawrapper + 0xbb00f)
                                            #2  0x000055e115bef080 _ZN9cCondWait7SleepMsEi (ffdecsawrapper + 0xbb080)
                                            #3  0x000055e115bbbf1a _ZN14cScHousekeeper6ActionEv (ffdecsawrapper + 0x87f1a)
                                            #4  0x000055e115beee75 _ZN7cThread11StartThreadEPS_ (ffdecsawrapper + 0xbae75)
                                            #5  0x00007f5755bff3e9 start_thread (libpthread.so.0 + 0x93e9)
                                            #6  0x00007f57557ee293 __clone (libc.so.6 + 0x100293)
Nov 26 12:34:35 test systemd[1]: systemd-coredump@9-577-0.service: Succeeded.

this is from code https://github.com/masjerang/descrambler/tree/testing (but I have also the same trap on bas-t descrambler code and also on old vdr code.) After some investigations i discovered issue seems to be FFDECSA optimisation related cpu march selection. FFdecsa_optimizer script checks is compiler supporting march=native - and if it is - then script suggest to go with march=native. Issue is that this is compile time check - not run-time check and this will make run of descrambler failed when runtime cpu arch is different than compile-time cpu arch. Ideal solution will be to implement few (most frequently meet) precompiled optimised FFDCSA binaries, benchmark them at descrambler start and select fastest one to use. This will require probably not small amount of work... Simpler solution might be: ask at script start: "will descrambler also run on this compile machine?". If yes then we can go with march=native. If not we should go with more safe i.e. march=x86-64 Let me know what you think about above...

masjerang commented 3 years ago

Hi Piotr,

I'm using 5.8.0 kernel without any issues on descrambler and dvbloopback. I haven't tested on 5.9.x kernel. I can try to look into it. Are you able to provide some additional logging? I'm not an expert on CPU arch selection....