search

bas-t / descrambler

Standalone version of FFdecsawrapper
GNU General Public License v3.0
7 stars 11 forks source link

descrambler builds but crashes if GCC's -fstack-check flag is used #3

Closed Strunzdesign closed 8 years ago

Strunzdesign commented 8 years ago

Hi,

I had some problems today getting the latest version of descrambler up and running, until I realized that the stack smashing protector (SSP) of gcc creates executables that crash immediately after decoding starts. Consider this as a bug report or a warning to all users of a hardened gcc that have SSP enabled.

I'm using hardened gentoo linux here.

gcc-config -l

[1] x86_64-pc-linux-gnu-4.9.3 * [2] x86_64-pc-linux-gnu-4.9.3-hardenednopie [3] x86_64-pc-linux-gnu-4.9.3-hardenednopiessp [4] x86_64-pc-linux-gnu-4.9.3-hardenednossp [5] x86_64-pc-linux-gnu-4.9.3-vanilla

Usually my gcc is set to profile [1] that enables both position-independend executables (PIE) and the stack smashing protector (SPP). Compiling the descrambler succeeds with all profiles (1, 2, 3, 4, and 5), but only the profiles 3, 4, and 5 create an executable that does not crash with a segmentation fault. Furthermore, this is not a freshly introduced misbehavior: I tested and picked multiple random commits even from the previous ffdecsawrapper repo and all of them are affected.

How to reproduce:

GDB shows the following (I mangled the bytes to assure not to publish any keys here!) [...snip...] Oct 11 19:55:59.439 CAM(core.ecm): 0.1: found 186a(0000) (Nagra2) id 0000 with ecm 1d8e/80 (new) Oct 11 19:55:59.439 THREAD: logger stats thread started (pid=26311, tid=3564746602240) Oct 11 19:55:59.439 THREAD: logger 0 filter thread started (pid=26311, tid=3564737484544) Oct 11 19:55:59.439 CAM(core.ecm): 0.1: try system Cardclient (1843) id 0000 with ecm 1234 (cached) (pri=-15) Oct 11 19:55:59.495 CAM(core.au): 0: chain caid 1830 -> Nagra2(-10) [1003-82/fe] Oct 11 19:55:59.495 CAM(core.au): 0: chain caid 1843 -> Nagra2(-10) Cardclient(-15) [1005-82/fe] Oct 11 19:55:59.495 CAM(core.au): 0: chain caid 09c4 -> none available Oct 11 19:55:59.495 CAM(core.au): 0: chain caid 098c -> none available Oct 11 19:55:59.495 CAM(core.au): 0: chain caid 0648 -> Irdeto2(-8) Irdeto(-10) [1016-82/ff] Oct 11 19:55:59.495 CAM(core.au): 0: chain caid 1860 -> Nagra2(-10) [1007-82/fe] Oct 11 19:55:59.496 CAM(core.au): 0: chain caid 0650 -> Irdeto2(-8) Irdeto(-10) [1019-82/ff] Oct 11 19:55:59.496 CAM(core.au): 0: chain caid 186a -> Nagra2(-10) [1013-82/fe] Oct 11 19:55:59.496 CAM(core.au): 0: starting chain 1843 Oct 11 19:55:59.614 CAM(cardclient.core): cc-loop Oct 11 19:55:59.614 CAM(cardclient.core): now trying client Newcamd (127.0.0.1:12345) Oct 11 19:55:59.987 CAM(core.ecm): 0.1: correct key found Oct 11 19:56:00.012 CSA: Creating csa for rb: 4 [New LWP 26360]

Program received signal SIGSEGV, Segmentation fault. [Switching to LWP 26360] 0x0000007599ec18bc in block_decypher_group(long long _vector, unsigned char_, unsigned char*, int) () (gdb) (gdb) (gdb) (gdb) bt

0 0x0000007599ec18bc in block_decypher_group(long long _vector, unsigned char_, unsigned char*, int) ()

1 0x0000007599ec6df5 in decryptpackets(void, unsigned char_*) ()

2 0x0000007599e9788b in process_ts (csa=0x759c935340,

buffer=0x33e3f1590a4 "G\001\002\003\004V>\004\006\007\008\009\010\011\012\013\014\015\016\...", end=129344, force=0) at dvblb_plugins/plugin_ffdecsa.c:327

3 0x0000007599e983b4 in process_ffd (msg=0x759c8ff890, priority=1) at dvblb_plugins/plugin_ffdecsa.c:512

4 0x0000007599e8c0b6 in msg_loop (arg=0x1) at src/msg_passing.c:118

5 0x0000033e421fa47c in ?? ()

6 0x0000000000000000 in ?? ()

(gdb)

Each packet of my system was compiled with "-ggdb", so backstraces normally contain useful information even regarding calls to system libraries.

Valgrind terminates with the following error: ==28158== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==28158== Bad permissions for mapped region at address 0x417A460 ==28158== at 0x1E68BC: block_decypher_group(long long _vector, unsigned char_, unsigned char*, int) (in /usr/bin/ffdecsawrapper)

Unfortunately, valgrind shows subsequent warnings before the crash and I would like to attach the whole dump it here, but (a) I'm not sure if that helps you and (b) I would like not to publish dumps that may contain "problematic numbers".

If I can help you / debug something... don't hesitate to reply :-) Florian

bas-t commented 8 years ago

Thanks for your report. I'll try to find some time to recompile my gcc/g++ with ssp (if it does not have it enabled, I don't know). I use Debian stable, so I get precompiled packages.

bas-t commented 8 years ago

Hi Florian, I read in this gentoo wiki:

https://wiki.gentoo.org/wiki/Hardened/FAQ#Why_don.27t_my_programs_work_when_I_use_CFLAGS.3D.22-O3.22_and_hardened_gcc.3F

that hardened gcc has problems with the -O3 flag. In fact it is not supported in hardened gcc.

The ffdecsawrapper part of the programm defaults to -O3

What happens when you compile descrambler, using profile 1 or 2, with -O2?

Strunzdesign commented 8 years ago

Hi bas-t,

thank you very much for your replies! Your link looks very interesting, but unfortunately, it did not help: I compiled the binary using "./configure --optimize=O2", installed and started it as usual, but it crashed again. A session with gdb shows exactly the same crash behavior.

I checked the output during compile to assure that the binary is compiled with -O2. It was.

Thank you very much! I'm looking forward assisting you further with that issue :-) Florian

bas-t commented 8 years ago

Hm, strange... I tried to trigger the segfault, compiling with -fPIC and -fstack-protector, but I cannot reproduce it. I use Irdeto2 though. Are those flags right? I mean does it look like what happens when using gentoo profile 1?

bas-t commented 8 years ago

Hi Florian, still trying to trigger the segfault.

Still no dice, everything is working as it should. Please show me exactly what compiler flags I have to use to trigger the segfault, so I can determine whether the root cause of it is in the FFdecsa part or maybe the nagra client part.

Cheers, Tycho.

Strunzdesign commented 8 years ago

Hi,

I'm currently testing, but a compiled binary still works with SSP disabled by profile BUT with -fstack-protector-strong enabled by the ffdescsawrapper-configure-buildscript.

Either my x86_64-pc-linux-gnu-4.9.3-hardenednossp overrides -fstack-protector-strong and thus creates a valid binary, or this is something else. I'm going to modify the build scripts now and exchange -fstack-protector-strong with -fno-stack-protector... lets see whether it compiles with the profile-enabled SSP then :-)

I'm still not sure how to query my system to obtain the default set of compile flags of my gcc. All I found so far is a file with gcc specs "hardenednossp.specs" containing the lines "_esp_cc1_ssp: " and "_esp_options_ssp:". No idea whether there is a set of default switches that enable some weird set of flags such as -fstack-protector and -fstack-protector-strong.

More info soon!

Strunzdesign commented 8 years ago

Ok, interesting information: the only relevant file is FFdecsa/FFdecsa.o!

If I compile everything with SSP enabled (profile [1] x86_64-pc-linux-gnu-4.9.3) then it creates a crashing binary. Then I remove file "FFdecsa/FFdecsa.o", switch to no-SSP-profile " [4] x86_64-pc-linux-gnu-4.9.3-hardenednossp", and type "make" again. It rebuilds the deleted object and links the ffdecsabinary, which does not crash :-D. If I switch back to SSP, remove FFdecsa.o again, type make, and install the freshly created binary again, the crash is back.

So, it seems that FFdecsa/FFdecsa.o contains the interesting code :-)

How to proceed now? Any idea how to query my gcc for its default flags?

Regards, Florian

Strunzdesign commented 8 years ago

No outcome editing config.mak to get FFdecsa.o built with -O1 and -fno-stack-protector... compiling and linking succeeds but the resulting binary still crashes :-( Seems that gcc adds some obscure flags internally, or overrides my explicit wish to disable stack protection.

Strunzdesign commented 8 years ago

GOT IT :-)

I'm able to reproduce it now with the "vanilla profile": just add "-fstack-check=specific":

g++ -O3 -march=native -fPIC -fstack-protector-strong -fexpensive-optimizations -fomit-frame-pointer -funroll-loops -fstack-check=specific -DPARALLEL_MODE=PARALLEL_128_SSE2 -c FFdecsa.c

This causes the bug here.

Ok, enough for today. Good luck! :-) Florian

bas-t commented 8 years ago

Thanks a lot. I'm not familiar with that flag, so I need to do some research about it before using it. That's not going to happen today. Sleep well.

joeusercz commented 8 years ago

Sorry I did not see this earlier, otherwise I could have suggested that there is a 99.9% chance the problem is with the ffdecsa code. In my work with oscam-emu/OSEmu I was testing many different compiler settings trying to optimize the performance of ffdecsa with various cross-compilers. While it was rare to get a segfault, quite often the code produced incorrect results. The code was written a long time ago and takes a lot of "shortcuts" to boost performance for the older x86 machines. These days it is probably not necessary given the power of the average cpu. (Although very helpful with low end set top boxes...)

Have you though about using libdvbcsa (from videolan.org) instead?

bas-t commented 8 years ago

@joeusercz: of what kind of incorrect results are you talking? and no, I never considered changing to libdvbcsa, since as it is it does an excellent job for me. However, if you want to (and libdvbcsa does not produce those incorrect results), feel free to implement it and send me a PR when you're done.

bas-t commented 8 years ago

@Strunzdesign : -fstack-check=specific is indeed preventing FFdecsawrapper from functioning. I did not yet read up on this flag, except for https://gcc.gnu.org/onlinedocs/gccint/Stack-Checking.html

Using the flag without the '=specific' part triggers

[ 457.146310] ffdecsawrapper[1860]: segfault at 7feb780644a8 ip 000000000049a19d sp 00007feb780642d0 error 6 in ffdecsawrapper[400000+e9000]

Do you have any pointers as to what I should read for a full understanding of this? Changing the title of your ticket.

bas-t commented 8 years ago

Hi Florian, please correct me if I'm wrong. It seems to me that since none of the STACK_CHECK macros are supported by the FFdecsawrapper code, the segfault you are encountering using the -fstack-check flag is not so much a bug in FFdecsa code, but rather a bug in gnu-gcc code. Imo, the compiler should check that used flags are actually supported by the code and if it finds lack thereof, either disregard the flag or print an error and exit.

What's your point of view?

@joeusercz : is this flag supported by libdvbcsa? I took a quick look into it, but did not find any evidence that it supports the flag. I may have overlooked something though.

Strunzdesign commented 8 years ago

Hi bas-t,

there is not many information available in the Internet describing the difference between -fstack-protect (which is mentioned to be equivalent to -fstack-protect=specific) and -fstack-protector(-strong). I read most of the links google provided to me, but there is not much new to tell you. As far as I understand, -fstack-protector and its derivatives works with canaries, but -fstack-protect works completely different and "simply" protects the stack from growing beyond some limits (e.g., due to recursion). However, most of the information seems to be outdated, and gcc switched its SPP implementation multiple times.

Regarding your last comment mentioning missing macros: I read that document too, but (a) I do not fully understand the purpose of these macros and (b) is -fstack-protect "the" compiler switch added to the default gcc spec on a box running the Gentoo hardened toolchain. That means, I was able to compile all of my approx. 1000 packets on my server using -fstack-protect, and I bet that most of the tarballs do not contain specifications regarding these macros. Nevertheless, enabling stack protection seems to be beneficial, not only for a small subset of the system. In my optionion (might be wrong!) the problem at hand is not caused by missing macros. Furthermore, I can compile all source files of ffdecsawrapper except one with -fstack-protect, link the binary, and it runs fine.

It would be interesting to use "c-reduce" to minimize the source code file to finally isolate the bug. If it is a compiler bug, this can result in a nice minimized example containing only the necessary stuff to trigger it. Perhaps I keep that for a little exercise.

How can I help you now? Florian

bas-t commented 8 years ago

Hi Florian, I think you misunderstood me. You are talking about the -fstack-protector flag, while I'm talking about the -fstack-check flag. Anyhow, you are probably right about most of the tarballs you compile don't have any stack_check macros defined. According to the official documentation, it should not only be defined but also contain code to check returns. I guess I'll have to make up my mind as to how I want to proceed. For now, I've simply overruled the -fstack-check flag by adding the -fno-stack-check flag to the compiler instructions.

Strunzdesign commented 8 years ago

Hi,

you are right, what I wrote in my last post was nonsense as I confused stack-check and stack-protect. To clarify, the hardened toolchain uses -fstack-check (with -fstack-check=specific being equivalent). However, you were able to decode my braindump correctly. Sorry!

BTW, I already checked out your last commit and compiled it, but I have to wait until a recording ends before I can test it. But I'm sure it is going to work well now.

This stack stuff has confusing names ;*) Florian

joeusercz commented 8 years ago

The incorrect results from ffdecsa were that the code did not decrypt properly the data. No errors were produced, just garbage output. I would like to try libdvbcsa, but it is not just a simple replacement of ffdecsa because the calls have different requirements and unfortunately I do not have the time now to play with it. Like ffdecsawrapper, for oscam-emu/OSEmu ffdecsa is working ok, so replacing it is a very low priority for me. I only asked because a few other projects have replaced ffdecsa with libdvbcsa and some like tsdecrypt have added ffdecsa now as an option after originally using libdvbcsa. It seems there are conflicting reports as to which is better, but that may be because one is better in certain situations and the other better in other cases...

As for the flag being supported, I am not sure because I have not actually used libdvbcsa yet..

Strunzdesign commented 8 years ago

Ok, I can confirm the the modified configure script works. With a gcc profile that enables SSP by default, the configure script successfully overrides it and disables the stack checker. Thanks :-)

bas-t commented 8 years ago

Hi Florian, thanks for the heads up. The way I see it, I've got two choices, one being adding STACK_CHECK defines and return checking code to accommodate gentoo users with hardened gcc profiles, the other one being just close this ticket as it works now out of the box. Hmm, lets see, that's a difficult one (not). Feel free to open a new ticket whenever you see fit, closing this one now. Cheers, Tycho.