search

avast / retdec

RetDec is a retargetable machine-code decompiler based on LLVM.
https://retdec.com/
MIT License
7.86k stars 936 forks source link

Big DLL can make retdec run out of stack #1206

Closed Delaunay closed 3 weeks ago

Delaunay commented 3 weeks ago

retdec crashes after sometime. I think it might have ran out of stack as the debugger cannot load the entire call stack.

Output

$ retdec-decompiler.exe lib_api.dll --no-memory-limit

Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `rdx_20_reload = rdx_20_reg2mem` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `r12_12_reload = r12_12_reg2mem` -> skipping this edge
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `break` -> skipping this edge
Running phase: signed/unsigned types fixing ( 1268.71s )

Call Stack

    ntdll.dll!00007ffe98d5b721()    Unknown
    ntdll.dll!00007ffe98d5b3c7()    Unknown
    ucrtbase.dll!00007ffe96bcfde6() Unknown
    retdec-decompiler.exe!operator new(unsigned __int64 size) Line 36   C++
    [Inline Frame] retdec-decompiler.exe!std::_Default_allocate_traits::_Allocate(const unsigned __int64) Line 90   C++
    [Inline Frame] retdec-decompiler.exe!std::_Allocate(const unsigned __int64) Line 248    C++
    [Inline Frame] retdec-decompiler.exe!std::allocator<std::_List_node<std::shared_ptr<retdec::llvmir2hll::Statement>,void *>>::allocate(const unsigned __int64) Line 982  C++
    [Inline Frame] retdec-decompiler.exe!std::_Alloc_construct_ptr<std::allocator<std::_List_node<std::shared_ptr<retdec::llvmir2hll::Statement>,void *>>>::_Allocate() Line 1160   C++
>   [Inline Frame] retdec-decompiler.exe!std::_List_node_emplace_op2<std::allocator<std::_List_node<std::shared_ptr<retdec::llvmir2hll::Statement>,void *>>>::{ctor}(std::allocator<std::_List_node<std::shared_ptr<retdec::llvmir2hll::Statement>,void *>> &) Line 584 C++
    retdec-decompiler.exe!std::_Hash<std::_Uset_traits<std::shared_ptr<retdec::llvmir2hll::Statement>,std::_Uhash_compare<std::shared_ptr<retdec::llvmir2hll::Statement>,std::hash<std::shared_ptr<retdec::llvmir2hll::Statement>>,std::equal_to<std::shared_ptr<retdec::llvmir2hll::Statement>>>,std::allocator<std::shared_ptr<retdec::llvmir2hll::Statement>>,0>>::emplace<std::shared_ptr<retdec::llvmir2hll::Statement> const &>(const std::shared_ptr<retdec::llvmir2hll::Statement> & <_Vals_0>) Line 610    C++
    [Inline Frame] retdec-decompiler.exe!std::_Hash<std::_Uset_traits<std::shared_ptr<retdec::llvmir2hll::Statement>,std::_Uhash_compare<std::shared_ptr<retdec::llvmir2hll::Statement>,std::hash<std::shared_ptr<retdec::llvmir2hll::Statement>>,std::equal_to<std::shared_ptr<retdec::llvmir2hll::Statement>>>,std::allocator<std::shared_ptr<retdec::llvmir2hll::Statement>>,0>>::insert(const std::shared_ptr<retdec::llvmir2hll::Statement> &) Line 931    C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visitStmt(std::shared_ptr<retdec::llvmir2hll::Statement> stmt, bool visitSuccessors, bool visitNestedStmts) Line 577   C++
    retdec-decompiler.exe!retdec::llvmir2hll::OrderedAllVisitor::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 122   C++
    retdec-decompiler.exe!retdec::llvmir2hll::ExprTypesAnalysis::visit(std::shared_ptr<retdec::llvmir2hll::AssignStmt> stmt) Line 188   C++
    retdec-decompiler.exe!retdec::llvmir2hll::AssignStmt::accept(retdec::llvmir2hll::Visitor * v) Line 165  C++
Delaunay commented 3 weeks ago

Adding the code below to the cmake recipe to see if I can bybass the issue with bigger stack.

if(MSVC)
    add_link_options(/STACK:4294967296)
else()
    add_link_options(-Wl,--stack,4294967296)
endif()
Delaunay commented 3 weeks ago

With the increased stack it was able to pass the problematic step