Closed havarnov closed 4 years ago
Here's the full "Problem Report" from macOS:
Thanks for the report!
Given your crash logs it says thread 20 crashed which has a stack trace of:
0 libwasmtime.dylib 0x0000000108cdb1a9 __rbt_macho_add_symtab + 1353
1 libwasmtime.dylib 0x0000000108cda230 __rbt_macho_add + 1920
2 libwasmtime.dylib 0x0000000108cd9921 __rbt_backtrace_initialize + 337
3 libwasmtime.dylib 0x0000000108cd8fdf fileline_initialize + 735
4 libwasmtime.dylib 0x0000000108cd90dd __rbt_backtrace_syminfo + 45
5 libwasmtime.dylib 0x0000000108cc4fcf backtrace::symbolize::libbacktrace::resolve::h719e8a9f1adb8fe1 + 159 (libbacktrace.rs:469)
6 libwasmtime.dylib 0x0000000108cc85a8 backtrace::symbolize::resolve_frame_unsynchronized::h7ca357de7ff98157 + 56 (mod.rs:178)
7 libwasmtime.dylib 0x0000000108cc83e0 backtrace::symbolize::resolve_frame::h75c5abc2a861fa2d + 80 (mod.rs:105)
8 libwasmtime.dylib 0x0000000108cc0c2c backtrace::capture::Backtrace::resolve::he887e05906dda0c3 + 348 (capture.rs:219)
9 libwasmtime.dylib 0x0000000108cc077c backtrace::capture::Backtrace::new::hde48dbbe07c85c38 + 60 (capture.rs:129)
10 libwasmtime.dylib 0x0000000108c6e4d0 wasmtime_runtime::traphandlers::Trap::wasm::hc86ae73cea48b813 + 64 (traphandlers.rs:363)
11 libwasmtime.dylib 0x0000000108c8bfa2 wasmtime_runtime::instance::initialize_memories::hffd4acb5f389bdf5 + 514 (instance.rs:1326)
12 libwasmtime.dylib 0x0000000108c89801 wasmtime_runtime::instance::InstanceHandle::new::h7769a621032ef434 + 6369 (instance.rs:978)
13 libwasmtime.dylib 0x00000001089ee425 wasmtime_jit::instantiate::CompiledModule::instantiate::h2f2ec277d7cd7968 + 1237 (instantiate.rs:216)
14 libwasmtime.dylib 0x00000001088d3286 wasmtime::instance::instantiate::hde8bac425f17f01c + 150 (instance.rs:29)
15 libwasmtime.dylib 0x00000001088d363a wasmtime::instance::Instance::new::hfb0a2766f04e3a70 + 586 (instance.rs:127)
16 libwasmtime.dylib 0x00000001088d11ee wasmtime::linker::Linker::instantiate::h4aa0fd178eec2ddd + 2878 (linker.rs:395)
17 libwasmtime.dylib 0x0000000108659214 wasmtime_linker_instantiate + 132 (linker.rs:85)
which looks like this is running into https://github.com/bytecodealliance/wasmtime/issues/1295 as an issue which is in turn slated to be fixed by https://github.com/bytecodealliance/wasmtime/pull/1315
NP, happy to help! Good that it's allready is on track to be fixed.
A couple of things:
var memory = new Memory(Store, 2, 2);
And I have a wasm module as follows:
(module
(import "" "mem" (memory $m 1 1))
)
Is there any way to grow the memory to 2 pages? I've tried to understand the spec, but haven't figured out how to use the memory.grow
instruction.
In the wasm module you defined the maximum size to be 1 page. This means that you can't grow it to 2 pages. If you use (memory $m 1)
instead, there is no limit on the size the memory can grow to other than the standard limit of 4GB.
@bjorn3 if you use (memory $m 1)
I guess you can't grow it more than what the host has defined? And how exactly do I use memory.grow
?
I believe you can use memory.grow
by first pushing a 32bit integer representing the new size in pages to the stack and the calling memory.grow
.
Ok I think this should be fixed with https://github.com/bytecodealliance/wasmtime/pull/1315 now
Still getting the same error after merging in master. Atleast for me it seems like it's the same stack trace (in thread 1 this time).
@havarnov that says that thread 1 crashed with a stack tha tlooks like:
Thread 1 Crashed:
0 libsystem_kernel.dylib 0x00007fff701817fa __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff7023ebc1 pthread_kill + 432
2 libsystem_c.dylib 0x00007fff70108a1c abort + 120
3 libcoreclr.dylib 0x000000010ce6815c SEHExceptionThread(void*) + 1884
4 libsystem_pthread.dylib 0x00007fff7023ee65 _pthread_start + 148
5 libsystem_pthread.dylib 0x00007fff7023a83b thread_start + 15
are you sure that's related to segfaults? That looks like the application explicitly killed itself?
Now I'm both confused and unsure about what a commented 2 days ago. Seems like a posted the wrong stack trace?
Anyways, I'm sure the test is failing/passing inconsistently. Here's a new stack trace after running dotnet test
on https://github.com/havarnov/wasmtime/commit/30e0655493c67cadf4099029e38fa726f896ea19.
Hm that also looks pretty normal, are you sure that was the thread which crashed?
I don't know how I manage to mess this up every time :/
I run dotnet test --filter ItThrowsOnOutOfBoundsMemoryAccess
.
It has three different outcomes:
Crashed thread stack trace:
Full report:
Hope I manage to get this corret now.
@havarnov ok that definitely looks like a bug! I don't think it's the same issue as before though because it doesn't look like there's any wasm code on the stack. This may perhaps mean there's a bug in the backtrace
crate or we're generating invalid debuginfo or something like that.
Would it be possible to reduce this to a small thing that can be reproduced locally?
@alexcrichton not sure what you mean about a "small thing"? Isn't the test I provided ok for testing? Or were you thinking of a small console application?
Hm so ideally we could get a reproduction with just the Rust API. Failing that a reproduction with just the C API would be great. Failing that a reproduction with a small example in the dotnet API would be great too. The dotnet extension has moved around a bit and it looks like the test mentioned here has either been renamed or no longer exists. Would it be possible to help reduce this to something standalone which can be reproduced today?
@alexcrichton I'll look into reproducing this bug in rust or the c api. The test I refered to was one I created in my fork of this repo, https://github.com/havarnov/wasmtime/tree/outofboundsmemoryaccess. That's why I asked if you wanted a PR with the new test.
I'll come back when I have something more to show, or I need more help. Ok?
@alexcrichton I need some help on this one. I'm trying to reproduce this by using the wasmtime api directly as follows, which to the best of my knowledge mimics the test I've written from the dotnet side.
#[test]
fn outofbounds_memory() -> Result<()> {
let store = Store::default();
let mut linker = Linker::new(&store);
let ty = MemoryType::new(Limits::new(1, Some(1)));
let memory = Memory::new(&store, ty);
let linker = linker.define("", "mem", memory).expect("Should be able to define memory item.");
let wat = r#"
(module
(import "" "mem" (memory $m 1 2))
(data (i32.const 65535) "\01")
(data (i32.const 65536) "\02")
)
"#;
let module = Module::new(&store, wat)?;
let trap= linker.instantiate(&module)
.err()
.unwrap()
.downcast::<Trap>()
.unwrap();
assert_eq!(
trap.message(),
"wasm trap: out of bounds memory access, source location: @-"
);
Ok(())
}
As you can see I'm expecting this to be "trapped" with the "out of bounds memory access" message. This is after all what I'm seeing from the dotnet side of things when the test does pass. But what I'm getting is a linker error from linker.instantiate
:
Failed to link module
Caused by:
Link error: memory out of bounds: data segment does not fit
I believe that's the expected error from that test. I don't really know much about C# testing, but you've got two tests defined in that file, is it possible that they're sharing access to the same linker by accident?
The ClearDefinitions
call should instantiate a new linker between the different test executions, I believe (iirc xUnit semantics correctly, at least).
Although, reusing the definitions seems like the probable cause of the failure. I can look into this.
I'm not able to reproduce the failure of either of those two tests (both pass as-is).
I verified that ClearDefinitions
is called for each test run, so a linker should not be shared between the runs.
@peterhuene interesting that it does fail on your machine. Just so there's no misunderstandings, it's the test name ItThrowsOnOutOfBoundsMemoryAccess
that's inconsistently passes.
I will try to get the "wasm trap: out of bounds memory access, source location: @-"
trap message from my rust test aswell, and see if that's working consistently.
I think I have pinpointed this a bit more. The ModuleFixture
in the dotnet test builds the Host as follows:
public ModuleFixture()
{
Host = new HostBuilder()
.WithMultiValue(true)
.WithReferenceTypes(true)
.Build();
Module = Host.LoadModuleText(Path.Combine("Modules", ModuleFileName));
}
EDIT: but to investigate this further I need some help from you guys; @alexcrichton @peterhuene
If I comment out .WithReferenceTypes(true)
the test now fails consistently with the following error message:
Error Message:
Expected exception message to match the equivalent of
"wasm trap: out of bounds memory access, source location: @-", but
"Failed to link module
Caused by:
Link error: memory out of bounds: data segment does not fit" does not.
@havarnov that's correct, the ItThrowsOnOutOfBoundsMemoryAccess
test consistently passes for me. I reran the tests 100 times and they all succeeded.
My steps to reproduce:
git clone git@github.com:bytecodealliance/wasmtime-dotnet.git
cd wasmtime-dotnet
tests/OutOfBoundsMemoryAccessTests.cs
with the contents of this file.tests/Modules/OutOfBoundsMemoryAccess.wat
with the contents of this file.dotnet test
in a loop 100 times, breaking if failure.All test runs were successful.
I'll see if I can reproduce it with your older wasmtime
-based branch.
However, I was able to reproduce the link error if support for reference types is disabled.
Expected a <Wasmtime.TrapException> to be thrown, but found <Wasmtime.WasmtimeException>: "
"Wasmtime.WasmtimeException with message "Failed to link module
Caused by:
Link error: memory out of bounds: data segment does not fit
It appears to be a link-time error without reference types enabled by design.
This is because the reference types feature implicitly enables the bulk-memory feature. Without it, you'll get the linker error.
I've added two tests for out of bounds memory access to WasmTime.Net. See: https://github.com/havarnov/wasmtime/commit/1b199cf63ed218094271020c34c89c2aea347dc3.
The problem is that one of the test ("ItThrowsOnOutOfBoundsMemoryAccess") is not consistently passing on my machine. Some times it's passing, sometimes the dotnet runtime crashes and sometimes the process hangs.