search

James-Jones / HLSLCrossCompiler

Other
468 stars 81 forks source link

Crash in FreeGLSLShader() #19

Closed nosferalatu closed 10 years ago

nosferalatu commented 10 years ago

I'm crashing in FreeGLSLShader() when it frees the psShaderInfo->psConstantBuffers variable. Specifically, the Visual Studio C runtime says that "this may be due to a corruption of the heap". In my code, I've hooked the malloc/calloc/free/realloc callbacks, but they just call directly to the C runtime.

I then built and ran the standalone hlslcc.exe executable, and that has the same issue with the same shader. It does work fine with a very simple test shader though.

I printed out all the calls to the hooked memory functions, and I see it allocate the same memory that it's later trying to free, and there's no double-free. Are there any known issues with memory corruption in the library?

I'll keep digging into it on my end. Is there a way to get you the shader binary (or even the source HLSL code, if that's helpful) that's causing the crash?

nosferalatu commented 10 years ago

Oh, I posted too soon. After digging in, the issue is that we have 387 uniform parameters. They aren't in a constant buffer (we're transitioning away from DX9) so HLSLCC was putting them in the global cb. However, MAX_SHADER_VAR = 256, and HLSLCC was writing off the end of the constant buffer array.

I changed MAX_SHADER_VAR = 1024 and everything worked fine. I think that constant buffers have a max size of 64K, but an unlimited number of parameters, as long as they all fit into that 64K.

James-Jones commented 10 years ago

I have removed the limitation on the number of constants allowed inside a constant buffer.