streamingfast / substreams

Powerful Blockchain streaming data engine, based on StreamingFast Firehose technology.
Apache License 2.0
164 stars 45 forks source link

Tier2 jobs get stuck on a Mac m1 after a while #224

Closed sduchesneau closed 1 year ago

sduchesneau commented 1 year ago

This probably happens on tier1 jobs too, but I've seen it more often on tier2 jobs, while running locally on my mac. This behavior is not seen in our production setup on linux hosts.

What to look for in a stacktrace (I get it by looking at http://localhost:6060/debug/pprof/goroutine?debug=2)

goroutine 127028 [syscall]:
github.com/bytecodealliance/wasmtime-go/v4._Cfunc_wasmtime_linker_delete(0x60000a650000)
    _cgo_gotypes.go:2287 +0x34
github.com/bytecodealliance/wasmtime-go/v4.(*Linker).FreeMem.func1.1(0x140015e0608?)
    /Users/stepd/go/pkg/mod/github.com/streamingfast/wasmtime-go/v4@v4.0.0-freemem3/linker.go:31 +0x50
github.com/bytecodealliance/wasmtime-go/v4.(*Linker).FreeMem.func1()
    /Users/stepd/go/pkg/mod/github.com/streamingfast/wasmtime-go/v4@v4.0.0-freemem3/linker.go:31 +0x20
sync.(*Once).doSlow(0x1080c0b20?, 0x14001b60050?)
    /opt/homebrew/Cellar/go/1.20.4/libexec/src/sync/once.go:74 +0x104
sync.(*Once).Do(...)
    /opt/homebrew/Cellar/go/1.20.4/libexec/src/sync/once.go:65
github.com/bytecodealliance/wasmtime-go/v4.(*Linker).FreeMem(0xffffffffffffffff?)
    /Users/stepd/go/pkg/mod/github.com/streamingfast/wasmtime-go/v4@v4.0.0-freemem3/linker.go:30 +0x48
github.com/streamingfast/substreams/wasm.(*Instance).FreeMem(0x14000860310)
    /Users/stepd/go/pkg/mod/github.com/streamingfast/substreams@v1.1.3/wasm/instance.go:33 +0x34

(...)

It also happens on feature/wazero branch when using SUBSTREAMS_WASM_RUNTIME=wasmtime

sduchesneau commented 1 year ago

Fixed https://github.com/streamingfast/substreams/pull/229 (wasmtime not used anymore)