tendermint / go-amino

Protobuf3 with Interface support - Designed for blockchains (deterministic, upgradeable, fast, and compact)
Other
259 stars 78 forks source link

Further investigate recursion depth and associated mem-consumption #211

Open liamsi opened 6 years ago

liamsi commented 6 years ago

Not too long ago the code snippet below produced a 100KB JSON message which consumed about 1,3 Gb memory while decoding.

package main
import (
"strings"
"github.com/tendermint/go-amino"
)
type Tree struct {
    Value int  `json:"v"`
    Children []Tree `json:"c"`
}
func main() {
    approxTargetLen := 100000
    msgBegin := `{"v":1,"c":[`
    msgEnd := `]}`
    repeatCnt := int(approxTargetLen / (len(msgBegin) + len(msgEnd)))
    recurMsg := strings.Repeat(msgBegin,repeatCnt )
    recurMsg = strings.Join([]string{recurMsg,strings.Repeat("]}",repeatCnt)},"")
    cst := Tree{}
    cdc := amino.NewCodec()
    cdc.UnmarshalJSON([]byte(recurMsg), &cst)
}

Although this is not the case anymore (it only consumes 45 Mb), we should make sure that we either use less recursion or limit the recursion depth (or fully understand why this isn't necessary anymore).

ValarDragon commented 6 years ago

ref #197