edrickhong / Cu

Game engine I am working on my free time
MIT License
2 stars 0 forks source link

MetaCall bugs in Windows #18

Closed edrickhong closed 6 years ago

edrickhong commented 6 years ago

The Windows implementation is buggy. Already found an issue with calling printf from a REFL function

void TEST(u32 a,f32 b,u32 c,f32 d){ printf("%d %f %d %f\n",a,b,c,d); }

This will crash inside the printf call if: only the lower 32 bits in the floating point values are written in xmm register. Works when the upper 32 bits are written

or if the arguments used f64 instead: the function will crash regardless

edrickhong commented 6 years ago

We'll investigate using ollydbg. We are however, not creating a frame pointer. We should probably do that.

edrickhong commented 6 years ago

Possible cause, the stack is not 16 byte aligned. Call pushes ret onto the stack which causes unalignment: 4 x 8 bytes for arg registers + 8 byte return address. Setting up a stack frame kinda does this:

Push rbp Move rbp,rsp Call Pop rbp

edrickhong commented 6 years ago

https://www.gamasutra.com/view/news/178446/Indepth_Windows_x64_ABI_Stack_frames.php

edrickhong commented 6 years ago

X64 stack layout:

Caller Non volatile regs Caller Local vars

Callee stack args Callee register args Ret address Rbp