vtil-project / VTIL-Core

Virtual-machine Translation Intermediate Language
BSD 3-Clause "New" or "Revised" License
1.35k stars 163 forks source link

Stack overflow on test_access #78

Open xtremegamer1 opened 2 years ago

xtremegamer1 commented 2 years ago

stack overflow

Here it is without the annoying function_view stuff obscuring the call stack:

without invocable

I will update the issue with more information and code snippets as I acquire it. This is during a dead code elimination pass. BTW the reason I say it is on test_access and not on tracer is that the stack overflow starts with a call to test_access but maybe is more accurate to say the stack overflow is in tracer? Kind of a chicken and egg situation

UPDATE: rtrace output

It looks like the same 3 symbolic variables are being repeatedly traced and that the failure is happening in enum_paths. Tracing one of the symbolic variables is causing enum_paths to recurse back into rtrace_primitive, tracing the same variable we started with, ad infinitum. I am guessing this has something to do with the linkage between these variables, maybe my input vtil is invalid? I will check on it.

Small update: It looks one expression has its paths enumerated, and two paths are traced, and the second path links back up to the first one causing infinite recursion. The one preceding "Enumerating paths" followed immediately by "done enumerating paths" is totally irrelevant as it is traced without incident.

xtremegamer1 commented 2 years ago

This might be related to #52 also sorry for opening 2 issues. I think there is probably multiple overflows on tracer

mrexodia commented 1 year ago

Currently VTIL isn’t actively maintained. If you could send a PR with a “minimal” (eg not 20mb) test vtil file and the code to reproduce the crash that would be appreciated.

xtremegamer1 commented 1 year ago

K I will send a pr and a minimum reproducible sample sometime in the next more than a week.

xtremegamer1 commented 1 year ago

It's a shame that most software deobfuscation tools are private. It really makes you feel like the assholes who slap vmprotect on their software and call it good are winning you know? It would be nice if VTIL worked right so people would need only to concern themselves with the (relatively) simple process of lifting and let the VTIL toolchain do the rest (right now it is lacking validation, reliable optimization, and compilation to amd64 machine code). I think this project hasn't really accomplished what it set out to but all its contributors got busy with other shit and moved on. Damn, I wish I could just code full time and wasn't busy with my shitty job, maybe I gotta enroll in college in some easy shit like calc 3 so i can quit my job and have an excuse to freeload off my parents after I'm 18. Then I can code all I want for a few years and have made enough off p2c after so I can buy my own place. Goddamnit why did I have to go and get myself kicked out of high school. I should have appreciated what I had, 7 hours a day to fuck around on my laptop pretending to work, now I gotta actually do work cause I have a boss now. Damn I just don't have the mental energy after 8 hours of moving heavy shit to track down a stack overflow especially with the extreme use of invokables and lambdas that fucks up the call stack. My life is so shit right now.

notphage commented 1 year ago

It's a shame that most software deobfuscation tools are private. It really makes you feel like the assholes who slap vmprotect on their software and call it good are winning you know? It would be nice if VTIL worked right so people would need only to concern themselves with the (relatively) simple process of lifting and let the VTIL toolchain do the rest (right now it is lacking validation, reliable optimization, and compilation to amd64 machine code). I think this project hasn't really accomplished what it set out to but all its contributors got busy with other shit and moved on. Damn, I wish I could just code full time and wasn't busy with my shitty job, maybe I gotta enroll in college in some easy shit like calc 3 so i can quit my job and have an excuse to freeload off my parents after I'm 18. Then I can code all I want for a few years and have made enough off p2c after so I can buy my own place. Goddamnit why did I have to go and get myself kicked out of high school. I should have appreciated what I had, 7 hours a day to fuck around on my laptop pretending to work, now I gotta actually do work cause I have a boss now. Damn I just don't have the mental energy after 8 hours of moving heavy shit to track down a stack overflow especially with the extreme use of invokables and lambdas that fucks up the call stack. My life is so shit right now.

Seek therapy

xtremegamer1 commented 1 year ago

It's a shame that most software deobfuscation tools are private. It really makes you feel like the assholes who slap vmprotect on their software and call it good are winning you know? It would be nice if VTIL worked right so people would need only to concern themselves with the (relatively) simple process of lifting and let the VTIL toolchain do the rest (right now it is lacking validation, reliable optimization, and compilation to amd64 machine code). I think this project hasn't really accomplished what it set out to but all its contributors got busy with other shit and moved on. Damn, I wish I could just code full time and wasn't busy with my shitty job, maybe I gotta enroll in college in some easy shit like calc 3 so i can quit my job and have an excuse to freeload off my parents after I'm 18. Then I can code all I want for a few years and have made enough off p2c after so I can buy my own place. Goddamnit why did I have to go and get myself kicked out of high school. I should have appreciated what I had, 7 hours a day to fuck around on my laptop pretending to work, now I gotta actually do work cause I have a boss now. Damn I just don't have the mental energy after 8 hours of moving heavy shit to track down a stack overflow especially with the extreme use of invokables and lambdas that fucks up the call stack. My life is so shit right now.

Seek therapy

My probation officer says I have to go to therapy so I will. I don't need to be healthy mentally though, I need to be healthy financially so I can quit my job and code full time.

mdo992 commented 1 year ago

It's a shame that most software deobfuscation tools are private. It really makes you feel like the assholes who slap vmprotect on their software and call it good are winning you know? It would be nice if VTIL worked right so people would need only to concern themselves with the (relatively) simple process of lifting and let the VTIL toolchain do the rest (right now it is lacking validation, reliable optimization, and compilation to amd64 machine code). I think this project hasn't really accomplished what it set out to but all its contributors got busy with other shit and moved on. Damn, I wish I could just code full time and wasn't busy with my shitty job, maybe I gotta enroll in college in some easy shit like calc 3 so i can quit my job and have an excuse to freeload off my parents after I'm 18. Then I can code all I want for a few years and have made enough off p2c after so I can buy my own place. Goddamnit why did I have to go and get myself kicked out of high school. I should have appreciated what I had, 7 hours a day to fuck around on my laptop pretending to work, now I gotta actually do work cause I have a boss now. Damn I just don't have the mental energy after 8 hours of moving heavy shit to track down a stack overflow especially with the extreme use of invokables and lambdas that fucks up the call stack. My life is so shit right now.

Seek therapy

My probation officer says I have to go to therapy so I will. I don't need to be healthy mentally though, I need to be healthy financially so I can quit my job and code full time.

you need both