Memory leak with go-wasm-adapter/go-wasm

[BruceChoca]

Thanks for the bug report!

Describe the bug

I referenced "github.com/go-wasm-adapter/go-wasm" and updated it to support wasmer-go 1.0+. When I create an instance with bridge(go-wasm) and execute it in batches, the memory is ever increasing. I try to optimize go-wasm, but the effect is not good, the problem still exists. Can you help me locate the cause?

Steps to reproduce

1. Go to github.com/BruceChoca/go-wasm, See "TestDemo"

   
   package memory

import ( "context" "encoding/json" "errors" "io/ioutil" "log" "runtime" "strconv" "sync" "testing" )

const ( FAILURE = 500 SUCCESS = 200 )

type Response struct { Result int json:"status" Message string json:"message" Body []byte json:"body" }

func TestDemo(t *testing.T) { codebuf, _ := ioutil.ReadFile("./wasm/main.wasm")

r, err := Start("test", codebuf)
if err != nil {
    panic("Start Error")
}

// batch
count := 30000
wg := sync.WaitGroup{}
wg.Add(count)
for i := 0; i < count; i++ {
    go func(index int) {
        err := r.Invoke(index)
        if err != nil {
            println(err.Error())
        }
        wg.Done()
    }(i)
}

wg.Wait()

println("done")

r.runtime.Release()

runtime.GC()

select {}

}

type R struct { runtime Runtime lock sync.Mutex }

func Start(name string, codebuf []byte) (*R, error) { runtime, _ := RuntimeFromBytes("test", codebuf, nil)

// import Call
runtime.SetFunc("Sys_Call", func(args []interface{}) (i interface{}, e error) {
    // get params
    funcName, _ := String(args[0])
    byteArgs, _ := Bytes(args[1])

    byteData := [][]byte{}
    err := json.Unmarshal(byteArgs, &byteData)
    if err != nil {
        resByte, _ := json.Marshal(Response{
            Result:  FAILURE,
            Message: "args Unmarshal error, Error: " + err.Error(),
        })
        return FromBytes(resByte), nil
    }

    switch funcName {
    case "Logln":
        args := []interface{}{}
        json.Unmarshal(byteData[0], &args)
        log.Println(args...)
        return FromBytes(nil), nil
    default:
        log.Println(args...)
        return FromBytes(nil), nil
    }
})

// start
init := make(chan error)
ctx, cancF := context.WithCancel(context.Background())
runtime.CancF = func() {
    runtime.ClearInstance()
    cancF()
}

go runtime.Run(ctx, init)

return &R{
    runtime: *runtime,
}, <-init

}

func (r *R) Invoke(index int) error { r.lock.Lock() defer r.lock.Unlock()

res, err := r.runtime.CallFunc("logFunc", []interface{}{"test log " + strconv.Itoa(index)})
if err != nil {
    return errors.New("CallFunc Error: " + err.Error())
}
resByte, err := Bytes(res)
if err != nil {
    return errors.New("Response Convert Bytes Error: " + err.Error())
}

var response = Response{}
json.Unmarshal(resByte, &response)

println("invoke", index, ":", response.Result, response.Message)

return nil

}

2. Run TestDemo
3. See error
<img width="820" alt="image" src="https://user-images.githubusercontent.com/26396986/124758870-989a4180-df61-11eb-8867-f3096f13cca3.png">

GOOS=js GOARCH=wasm go build -o main.wasm ./wasm/main.go

### Expected behavior

I expect the memory usage to grown a bit and then even out to a stable number.

### Actual behavior

The memory usage keeps increasing with invoke.

[Hywan]

Hello,

So, you're compiling a Go program to Wasm, to execute it in Go. OK.

First, why doing that :-)? I'm curious. Second, maybe it's a leak in your Go program, not in wasmer-go. Can you try with https://github.com/wasmerio/wasmer-go/pull/277 please?

[BruceChoca]

Hello,

First, I want to make a tool to run go programs by wasm. Second, I think the leak may occur in the following parts:

1. Runtime

I update "syscall/js.finalizeRef" and optimized it.

func finalizeRef(env interface{}, v []wasmer.Value) (out []wasmer.Value, err error) {
    sp := v[0].I32()

    b := getRuntime(env)
    // log.Println("finalizeRef: ", len(b.valueMap), ", refs: ", len(b.refs), ", refcout: ", len(b.refCount))
    id := int(b.getUint32(sp + 8))

    b.valuesMu.RLock()
    b.refCount[id]--
    if b.refCount[id] == 0 {
        v, ok := b.valueMap[id]

        if ok {
            rt := reflect.TypeOf(v)
            if rt.Kind() == reflect.Ptr {
                rt = rt.Elem()
            }
            rv := v
            if !rt.Comparable() {
                rv = reflect.ValueOf(v)
            }

            delete(b.refs, rv)
            delete(b.valueMap, id)
            delete(b.refCount, id)
            // b.idPools = append(b.idPools, id)
        }
    }
    b.valuesMu.RUnlock()

    return
}

The end of demo, I run Release(), set them "nil"

func (b *Runtime) ClearInstance() {
    b.instance = nil
    b.valueMap = nil
    b.refs = nil
    b.valueIDX = 8
    b.refCount = nil
    b.memory = nil
}

2. Wasm Memory, I called "Grow"

   func (b *Runtime) mem(offset int32) []byte {
   m, _ := b.instance.Exports.GetMemory("mem")
   if b.memory == nil {
       b.memory = m.Data()
   }
   
   if len(b.memory) <= int(offset) {
       m.Grow(1)
       b.memory = m.Data()
   }
   
   return b.memory
   }

3. Wasmer-go

I can't judge the real cause. my go program referenced "github.com/go-wasm-adapter/go-wasm", I also tested "github.com/go-wasm-adapter/go-wasm", it also has similar problems. I saw this 8#issue go-wasm-adapter/go-wasm

So, can you help me find the reason?

Hello,

So, you're compiling a Go program to Wasm, to execute it in Go. OK.

First, why doing that :-)? I'm curious. Second, maybe it's a leak in your Go program, not in wasmer-go. Can you try with #277 please?

[Hywan]

I'm sorry but you should open a PR on go-wasm-adapter/go-wasm, not this repository. I'm not the owner of the go-wasm project. I proposed my help to migrate to wasmer-go 1.0, but the discussion must not happen here :-).

I'm closing the issue. Please reopen it at https://github.com/go-wasm-adapter/go-wasm/. Thank you.