RobinDavid
commented
4 years ago Hi Peter,
You are right. We are aware of this bias. Fortunately for EA this concerns a small number of cases (~20 or so) so there is very negligible impact on results. I don't think EA-VR and EA-ED have this issue. (we were already in the process of writing/submitting the paper when noticing it and re-running alls benchmarks was cumbersome). There is somewhat the same bias with Triton if by means of symbolic execution it comes up with an expression already synthesized. It would be considered synthesized even though the synthesizer itself have done nothing. We did not performed any statistics on that. I could rerun with O0 (when I will have some time) but changing the benchmarks would be a bad idea as it is better that other tools compares with the same benchmark (yet containing this small bias) rather recompiling another one for which other tools would not be able to fairly compare against results we published for qsynth.
Thanks for your feedback, (hopefully looking forward for your results on our (imperfect) benchmark :) )
Regards, Robin
Hi.
Some of the evaluations seem to be wrong as you compiled the obfuscated binary with compiler optimizations and the compiler was already able to optimize the expression.
Example:
[INFO] ================ Processing function target_2 ==================== [INFO] [ORIGINAL]: e + (c | c) - (c + e) | -(c & c)
[INFO] Orig:[sz:14, d:5] Obfu:[sz:65, d:13] Triton:[sz:2, d:2] Synthesized:[sz:2, d:2] [Simp:OK] [FullSynth:OK] [INFO] [EQUIVALENCE]: [org-obf:UNK] [obf-trit:UNK] [org-final:OK]
Assembler code of function in the obfuscated binary:
.text:0000000000423AA4 push rbp .text:0000000000423AA5 mov rbp, rsp .text:0000000000423AA8 mov [rbp+var_8], rdi .text:0000000000423AAC mov [rbp+var_10], rsi .text:0000000000423AB0 mov [rbp+var_18], rdx .text:0000000000423AB4 mov [rbp+var_20], rcx .text:0000000000423AB8 mov [rbp+var_28], r8 .text:0000000000423ABC mov rax, [rbp+var_18] .text:0000000000423AC0 neg rax .text:0000000000423AC3 pop rbp .text:0000000000423AC4 retn
As you can see the compiler already optimized 'e + (c | c) - (c + e) | -(c & c)' to '-c' so there is nothing to be synthesized. It seems that this happens to all expressions even with tigress obfuscation where the expressions are simplified. I think it would be the good if you could recompile the sample with "O0" and run the evaluation again.
Peter