tomaka / hlua

Rust library to interface with Lua
MIT License
510 stars 48 forks source link

Potentially unaligned userdata pointer causes segfault #219

Closed vallentin closed 5 months ago

vallentin commented 5 months ago

I'm unsure if this is platform specific, since I was confused why it hadn't been reported yet.

I attempted to create a namespace with methods for a custom type. However, calling any method immediately segfaulted. I thought I had a typo somewhere, but when I ran the sound-api.rs example (on Windows 10), using hlua = "0.4.2" it also segfaulted:

   Compiling lua v0.1.0 (C:\Users\vallentin\Desktop\lua)
    Finished dev [unoptimized + debuginfo] target(s) in 0.40s
     Running `target\debug\lua.exe`
error: process didn't exit successfully: `target\debug\lua.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
Segmentation fault

I tested the Lua code line by line, and the segfault is caused by s:play(), or more precisely by calling any method, e.g. s:play(), s:is_playing(), s:stop().

I also noted that if only Sound.new() is executed, then it successfully runs the Sound destructor, i.e. hlua calling drop internally does not result in a segfault.


I went digging a bit, and hlua = "=0.3.1" does not segfault, while the next version hlua = "=0.4.0" does.

Digging a bit further, and https://github.com/tomaka/hlua/commit/b38af4200466f278850171bdff7cc74658f0f92b is the commit introducing the segfault, while the previous commit https://github.com/tomaka/hlua/commit/634a4dee5a0d1d0418287b95dfa77a524b502862 does not cause any segfault.

i.e. first version causing segfault:

hlua = { git = "https://github.com/tomaka/hlua", rev="b38af4200466f278850171bdff7cc74658f0f92b" }

vs last good version not causing a segfault:

hlua = { git = "https://github.com/tomaka/hlua", rev="634a4dee5a0d1d0418287b95dfa77a524b502862" }

I didn't dig any further than that so far. I'll happily provide any other information needed.

vallentin commented 5 months ago

I cloned the repo and ran cargo test and multiple_userdata() in userdata.rs segfaulted.

https://github.com/tomaka/hlua/blob/94f4a80fb85aa846e8a15dce9f23db57d932f5f2/hlua/tests/userdata.rs#L160-L277

Removing all asserts at the end of the test, and there's no segfault, but simply executing one / the first, triggers a segfault. So the segfault is first triggered when executing Lua code.

This seems to be further proved, by the segfault being triggered in call_with_args() by ffi::lua_pcall().


_Still in the multiple_userdata() (userdata.rs) test._

If we remove the custom Integer type, then the following passes:

lua.set("add", hlua::function0(|| 123));
assert_eq!(
    lua.execute::<u32>("return add()").unwrap(),
    123
);

While if we wrap it in Integer, then it segfaults:

lua.set("add", hlua::function0(|| Integer(123)));
assert_eq!(
    lua.execute::<Integer>("return add()").unwrap().0,
    123
);

Clearly, it has something to do with resolving userdata back into Rust types.


I found the culprit. It seems that the pointer returned by lua_newuserdata might be unaligned. Which triggers the segfault when it is dereferenced:

misaligned pointer dereference: address must be a multiple of 0x10 but is 0x22c615c7ef8

https://github.com/tomaka/hlua/blob/94f4a80fb85aa846e8a15dce9f23db57d932f5f2/hlua/src/userdata.rs#L139

I will write a PR to resolve the issue.