tetratelabs / wazero

wazero: the zero dependency WebAssembly runtime for Go developers
https://wazero.io
Apache License 2.0
4.97k stars 259 forks source link

Looking up and calling a non existing function will SIGILL on some platforms #1621

Closed jerbob92 closed 1 year ago

jerbob92 commented 1 year ago

Describe the bug This is a result of the issue we found when developing #1611, that code would try to call the exported function stackRestore, but that function was not defined in our WASM, because we didn't expect the current tests to reach that code. On some platforms this will SIGILL: illegal instruction some of the times, at least on arm64 and riscv64.

To Reproduce First make sure you can run arm64 in Docker by install qemu:

docker run --privileged --rm ghcr.io/tetratelabs/wazero/internal-binfmt --install arm64

Create a Dockerfile with the following:

FROM scratch
CMD ["/test"]

Build the docker image:

docker buildx build -t wazero:test --platform linux/arm64 .

Create the following main.go:

package main

import (
    "context"
    _ "embed"
    "os"

    "github.com/tetratelabs/wazero"
    "github.com/tetratelabs/wazero/experimental"
    "github.com/tetratelabs/wazero/experimental/logging"
    "github.com/tetratelabs/wazero/imports/emscripten"
)

// This is the invoke.wasm from imports/emscripten/testdata.
//
//go:embed invoke.wasm
var invokeWasm []byte

func main() {
    ctx := context.WithValue(context.Background(), experimental.FunctionListenerFactoryKey{}, logging.NewLoggingListenerFactory(os.Stdout))

    r := wazero.NewRuntime(ctx)
    defer r.Close(ctx)

    compiled, err := r.CompileModule(ctx, invokeWasm)
    if err != nil {
        panic(err)
    }

    _, err = emscripten.InstantiateForModule(ctx, r, compiled)
    if err != nil {
        panic(err)
    }

    mod, err := r.InstantiateModule(ctx, compiled, wazero.NewModuleConfig())
    if err != nil {
        panic(err)
    }

    mod.ExportedFunction("iDontExist").Call(ctx)
}

Build the binary and run it in Docker:

export GOARCH=arm64
export CGO_ENABLED=0
go build -o reproduce main.go
docker run --platform linux/arm64 -v $(pwd)/reproduce:/test --tmpfs /tmp --rm -t wazero:test

You might have to run it a few times to trigger it. If you first check the exported function for nil it doesn't have this behaviour.

Expected behavior ExportedFunction should always return nil when the function doesn't exist.

Environment (please complete the relevant information):

Additional context

SIGILL: illegal instruction
PC=0x0 m=0 sigcode=2
instruction bytes: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

goroutine 1 [running]:
main.main()
    /home/jeroen/Projects/OpenSource/wazero/reproduce/main.go:42 +0x338 fp=0x4000173f20 sp=0x4000173e30 pc=0x1b34b8
runtime: g 1: unexpected return pc for main.main called from 0x400000e018
stack: frame={sp:0x4000173f20, fp:0x4000174010} stack=[0x400016c000,0x4000174000)
0x0000004000173e20:  0x000000400014a500  0x000000400005de78 
0x0000004000173e30:  0x00000000001b34b8 <main.main+0x0000000000000338>  0x000000400014a500 
0x0000004000173e40:  0x0000000000220af2  0x000000400005de78 
0x0000004000173e50:  0x00000000001b3444 <main.main+0x00000000000002c4>  0x0000000000000010 
0x0000004000173e60:  0x00000000001db540  0x0100000000000001 
0x0000004000173e70:  0x0000000000230ad8  0x000000400004c768 
0x0000004000173e80:  0x00000000000454a0 <runtime.main+0x0000000000000200>  0x0000000000268148 
0x0000004000173e90:  0x000000400014a500  0x0000000000266160 
0x0000004000173ea0:  0x000000400007da40  0x0000004000013440 
0x0000004000173eb0:  0x0000000000268a00  0x000000400018a000 
0x0000004000173ec0:  0x0100000000014904  0x0000000000265ce8 
0x0000004000173ed0:  0x0000000000000001  0x0000004000013300 
0x0000004000173ee0:  0x000000400007db30  0x000000400014a500 
0x0000004000173ef0:  0x000000400007da40  0x0000004000013440 
0x0000004000173f00:  0x0000000000268148  0x0000000000266160 
0x0000004000173f10:  0x0000004000129000  0x0000000000267078 
0x0000004000173f20: <0x000000400000e018  0x0000004000014030 
0x0000004000173f30:  0x000000400018a000  0x00000000001b3540 <main.main.func1+0x0000000000000000> 
0x0000004000173f40:  0x0000000000267078  0x000000400007db30 
0x0000004000173f50:  0x0000000000266160  0x000000400007da40 
0x0000004000173f60:  0x0000004000173f38  0x0000000000000000 
0x0000004000173f70:  0x0000000000072024 <runtime.goexit+0x0000000000000004>  0x000000400007a000 
0x0000004000173f80:  0x0000000000000000  0x0000000000000000 
0x0000004000173f90:  0x0000000000000000  0x0100000000000000 
0x0000004000173fa0:  0x0000000000000000  0x00000000003a3b20 
0x0000004000173fb0:  0x00000000000455d0 <runtime.main.func2+0x0000000000000000>  0x0000004000173f9e 
0x0000004000173fc0:  0x0000004000173fb0  0x0000000000000000 
0x0000004000173fd0:  0x0000000000000000  0x0000000000000000 
0x0000004000173fe0:  0x0000000000000000  0x0000000000000000 
0x0000004000173ff0:  0x0000000000000000  0x0000000000000000 
main.main()
    /home/jeroen/Projects/OpenSource/wazero/reproduce/main.go:43 +0x338 fp=0x4000174010 sp=0x4000173f20 pc=0x1b34b8

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x400004cfa0 sp=0x400004cf80 pc=0x458b0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0x400004cfd0 sp=0x400004cfa0 pc=0x45700
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400004cfd0 sp=0x400004cfd0 pc=0x72024
created by runtime.init.6
    /usr/local/go/src/runtime/proc.go:293 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x400004d760 sp=0x400004d740 pc=0x458b0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:278 +0x98 fp=0x400004d7b0 sp=0x400004d760 pc=0x32328
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:178 +0x28 fp=0x400004d7d0 sp=0x400004d7b0 pc=0x273e8
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400004d7d0 sp=0x400004d7d0 pc=0x72024
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:178 +0x6c

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x4000072000?, 0x262fd0?, 0x1?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x400004df50 sp=0x400004df30 pc=0x458b0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x3a35c0)
    /usr/local/go/src/runtime/mgcscavenge.go:400 +0x5c fp=0x400004df80 sp=0x400004df50 pc=0x302bc
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:628 +0x44 fp=0x400004dfb0 sp=0x400004df80 pc=0x307f4
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:179 +0x28 fp=0x400004dfd0 sp=0x400004dfb0 pc=0x27388
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400004dfd0 sp=0x400004dfd0 pc=0x72024
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:179 +0xac

goroutine 5 [finalizer wait]:
runtime.gopark(0x400004c5a8?, 0x60000000024e68?, 0x28?, 0xd7?, 0x1?)
    /usr/local/go/src/runtime/proc.go:381 +0xe0 fp=0x400004c580 sp=0x400004c560 pc=0x458b0
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x100 fp=0x400004c7d0 sp=0x400004c580 pc=0x264b0
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400004c7d0 sp=0x400004c7d0 pc=0x72024
created by runtime.createfing
    /usr/local/go/src/runtime/mfinal.go:163 +0x80

r0      0x0
r1      0x266160
r2      0x400007da40
r3      0x0
r4      0x0
r5      0x0
r6      0x0
r7      0x0
r8      0x10
r9      0x4000011d60
r10     0x0
r11     0x0
r12     0x1d6
r13     0x200
r14     0x1d7
r15     0x18
r16     0x400016c3a0
r17     0x4000173b80
r18     0x0
r19     0x0
r20     0x4000173b50
r21     0x0
r22     0x0
r23     0x0
r24     0x0
r25     0x0
r26     0x230eb0
r27     0x3d7000
r28     0x40000021a0
r29     0x4000173e28
lr      0x1b3630
sp      0x4000173e30
pc      0x0
fault   0x0
evacchi commented 1 year ago

thanks for the detailed reproducer! EDIT: confirmed on native arm64 (Apple M1) with any wasm file EDIT2: wait no. It just SIGSEGVs because ExportedFunction() returns nil 🤔 EDIT3: finally, managed to reproduce with linux/riscv64

❯ docker run --platform linux/riscv64 -v $(pwd)/repro:/test -v $(pwd)/repro:/wazero -e WAZEROCLI=/wazero --tmpfs /tmp --rm -t wazero:test ``` ❯ docker run --platform linux/riscv64 -v $(pwd)/repro:/test -v $(pwd)/repro:/wazero -e WAZEROCLI=/wazero --tmpfs /tmp --rm -t wazero:test SIGILL: illegal instruction PC=0x0 m=0 sigcode=1 instruction bytes: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 goroutine 1 [running]: runtime/internal/atomic.(*Uint32).Load(...) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/internal/atomic/types.go:194 runtime.main() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:260 +0x218 fp=0xc000129e18 sp=0xc000129dc0 pc=0x45348 runtime: g 1: unexpected return pc for runtime.main called from 0xc0000a3050 stack: frame={sp:0xc000129e18, fp:0xc000129e70} stack=[0xc000122000,0xc00012a000) 0x000000c000129d18: 0x0000000000000000 0x0000000000000001 0x000000c000129d28: 0x00000000001bff84 0x00000000001f3080 0x000000c000129d38: 0x00000000001e6940 0x000000c000144f10 0x000000c000129d48: 0x00000000001e6940 0x00000000003eff80 0x000000c000129d58: 0x000000c0001580f0 0x0000000000000000 0x000000c000129d68: 0x000000c000130800 0x00000000001f3080 0x000000c000129d78: 0x0000000000000000 0x0000000000000000 0x000000c000129d88: 0x0000000000000000 0x0000000000000000 0x000000c000129d98: 0x00000000001bffc8 0x000000c000130800 0x000000c000129da8: 0x0000000000225642 0x000000000000000a 0x000000c000129db8: 0x0000000000000000 0x0000000000045348 0x000000c000129dc8: 0x000000c000130800 0x0000000000225642 0x000000c000129dd8: 0x000000c0000a2f90 0x000000000026b3c8 0x000000c000129de8: 0x000000c0000ad580 0x000000000026b960 0x000000c000129df8: 0x000000c0001580f0 0x01000000003c3a40 0x000000c000129e08: 0x000000000026aaa0 0x000000c0000ad4c0 0x000000c000129e18: <0x000000c0000a3050 0x000000c0000a2f90 0x000000c000129e28: 0x000000c0000ad580 0x000000000026b3c8 0x000000c000129e38: 0x000000000026af18 0x000000c0000d5000 0x000000c000129e48: 0x000000000026b558 0x000000c0000a8008 0x000000c000129e58: 0x000000c0000a6000 0x00000000001c0080 0x000000c000129e68: 0x000000000026b558 >0x000000c0000a3050 0x000000c000129e78: 0x000000000026af18 0x000000c0000a2f90 0x000000c000129e88: 0x0000000000000000 0x0000000000000000 0x000000c000129e98: 0x0000000000000000 0x000000c000144ee0 0x000000c000129ea8: 0x0000000000000001 0x0000000000000001 0x000000c000129eb8: 0x0000000000000000 0x0000000000000000 0x000000c000129ec8: 0x0000000000000000 0x0000000000000000 0x000000c000129ed8: 0x0000000000000000 0x0000000000000000 0x000000c000129ee8: 0x0000000000000000 0x0000000000000000 0x000000c000129ef8: 0x0000000000000000 0x0000000000000000 0x000000c000129f08: 0x0000000000000000 0x0000000000000000 0x000000c000129f18: 0x0000000000000000 0x0000000000000000 0x000000c000129f28: 0x0000000000000000 0x0000000000000000 0x000000c000129f38: 0x0000000000000000 0x0000000000000000 0x000000c000129f48: 0x0000000000000000 0x0000000000000000 0x000000c000129f58: 0x000000c0001563c0 0x0000000000000000 0x000000c000129f68: 0x0000000000000000 runtime.main() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:250 +0x218 fp=0xc000129e70 sp=0xc000129e18 pc=0x45348 goroutine 2 [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:381 +0x110 fp=0xc00002cfb0 sp=0xc00002cf98 pc=0x457b8 runtime.goparkunlock(...) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:387 runtime.forcegchelper() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:305 +0xc8 fp=0xc00002cfd8 sp=0xc00002cfb0 pc=0x455f0 runtime.goexit() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/asm_riscv64.s:512 +0x4 fp=0xc00002cfd8 sp=0xc00002cfd8 pc=0x70e24 created by runtime.init.5 /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:293 +0x28 goroutine 3 [GC sweep wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:381 +0x110 fp=0xc00002d788 sp=0xc00002d770 pc=0x457b8 runtime.goparkunlock(...) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:387 runtime.bgsweep(0x0?) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgcsweep.go:278 +0xa8 fp=0xc00002d7c8 sp=0xc00002d788 pc=0x31958 runtime.gcenable.func1() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgc.go:178 +0x2c fp=0xc00002d7d8 sp=0xc00002d7c8 pc=0x26674 runtime.goexit() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/asm_riscv64.s:512 +0x4 fp=0xc00002d7d8 sp=0xc00002d7d8 pc=0x70e24 created by runtime.gcenable /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgc.go:178 +0x70 goroutine 4 [GC scavenge wait]: runtime.gopark(0xc00004a000?, 0x267e78?, 0x0?, 0x0?, 0x0?) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:381 +0x110 fp=0xc00002df80 sp=0xc00002df68 pc=0x457b8 runtime.goparkunlock(...) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:387 runtime.(*scavengerState).park(0x3c3540) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgcscavenge.go:400 +0x68 fp=0xc00002dfa8 sp=0xc00002df80 pc=0x2f830 runtime.bgscavenge(0x0?) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgcscavenge.go:628 +0x50 fp=0xc00002dfc8 sp=0xc00002dfa8 pc=0x2fd88 runtime.gcenable.func2() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgc.go:179 +0x2c fp=0xc00002dfd8 sp=0xc00002dfc8 pc=0x26614 runtime.goexit() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/asm_riscv64.s:512 +0x4 fp=0xc00002dfd8 sp=0xc00002dfd8 pc=0x70e24 created by runtime.gcenable /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mgc.go:179 +0xb0 goroutine 17 [finalizer wait]: runtime.gopark(0xc00002c590?, 0x11?, 0x0?, 0x45?, 0x219e40?) /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/proc.go:381 +0x110 fp=0xc00002c5a8 sp=0xc00002c590 pc=0x457b8 runtime.runfinq() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mfinal.go:193 +0x100 fp=0xc00002c7d8 sp=0xc00002c5a8 pc=0x25780 runtime.goexit() /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/asm_riscv64.s:512 +0x4 fp=0xc00002c7d8 sp=0xc00002c7d8 pc=0x70e24 created by runtime.createfing /opt/homebrew/Cellar/go/1.20.7/libexec/src/runtime/mfinal.go:163 +0x64 ra 0x1bffe8 sp 0xc000129dc0 gp 0x0 tp 0x0 t0 0x0 t1 0x0 t2 0x0 s0 0x0 s1 0xf28 a0 0x0 a1 0x26af18 a2 0xc0000a2f90 a3 0x0 a4 0x0 a5 0x0 a6 0x69569aaaa a7 0x2220656c75646f6d s2 0x25 s3 0x65 s4 0x78 s5 0x70 s6 0x62cb654a1bf5747a s7 0x4 s8 0xc0000fc2d0 s9 0xc000129b80 s10 0x2350d0 s11 0xc0000001a0 t3 0x0 t4 0x0 t5 0x0 t6 0x1ffff pc 0x0 ```

But I can confirm it's only a mismanaged nil ptr.

    f := mod.ExportedFunction("iDontExist")
    if f == nil {
        println("nope")
    } else {
        f.Call(ctx)
    }

prints nope.

EDIT4: @jerbob92 I think you found bug in the Go compiler

❯ cat reproducer.go
package main

type i interface {
    f()
}

func main() {
    var ii i
    ii.f()
}

⬇️

❯ GOARCH=riscv64 GOOS=linux go build -gcflags=all="-N -l" reproducer.go

❯ docker run --platform linux/riscv64 -v $(pwd)/reproducer:/test -v $(pwd)/reproducer:/wazero -e WAZEROCLI=/wazero --tmpfs /tmp --rm -t wazero:test

💣💥

mathetake commented 1 year ago

LOL this is nothing to do with wazero, but simply a bug of Go compiler or QEMU...

$ cat Dockerfile
FROM scratch
CMD ["/test"]

$ docker buildx build -t testriscv --platform linux/riscv64 .
$ cat main.go
package main

func main() {
    getter().Call()
}

type iface interface {
    Call()
}

func getter() iface {
    return nil
}

$ GOARCH=riscv64 GOOS=linux CGO_ENABLED=0 go build -o crash main.go 
$ docker run --platform linux/riscv64 -v $(pwd)/crash:/test --tmpfs /tmp --rm -t testriscv
SIGILL: illegal instruction
PC=0x0 m=0 sigcode=1
instruction bytes: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0

goroutine 1 [running]:
runtime/internal/atomic.(*Uint32).Load(...)
        /usr/local/go/src/runtime/internal/atomic/types.go:194
runtime.main()
        /usr/local/go/src/runtime/proc.go:260 +0x218 fp=0xc00002c7c8 sp=0xc00002c770 pc=0x3dcd8
runtime: g 1: unexpected return pc for runtime.main called from 0xc00002c7ae
---snip---
mathetake commented 1 year ago

Closing this as it's out of our hands, and I landed the workaround #1623 as far as we are concerned