fatal error: runtime: out of memory when trying to deserialize bad transactions

A somewhat interesting problem: we were running a service that takes in encoded transactions as user input, then deserializes them to do some things. We then saw a bunch of out of memory errors, which was weird because our application never got close to the limit we had set for it.

After a bit of tracing, we tracked down the issue to bad user input. Specifically, the current implementation of the transaction decoder (and possibly others) doesn't do super strong validation of the incoming data, and somewhat naively allocates slices based on certain byte offsets. For example, this was our problem line: https://github.com/gagliardetto/solana-go/blob/main/message.go#L452

See the runtime stack:

goroutine 16504 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:436 fp=0xc000886a60 sp=0xc000886a58 pc=0x46c660
runtime.(*mheap).alloc(0x163e65b0000?, 0xb1f32d8?, 0xf0?)
    /usr/local/go/src/runtime/mheap.go:906 +0x65 fp=0xc000886aa8 sp=0xc000886a60 pc=0x42c2c5
runtime.(*mcache).allocLarge(0x10ca340?, 0x163e65afc58, 0x0)
    /usr/local/go/src/runtime/mcache.go:213 +0x85 fp=0xc000886af8 sp=0xc000886aa8 pc=0x41a5e5
runtime.mallocgc(0x163e65afc58, 0xe84500, 0x1)
    /usr/local/go/src/runtime/malloc.go:1096 +0x5a5 fp=0xc000886b70 sp=0xc000886af8 pc=0x4109e5
runtime.makeslice(0x1?, 0xc000886bf8?, 0x41ab13?)
    /usr/local/go/src/runtime/slice.go:103 +0x52 fp=0xc000886b98 sp=0xc000886b70 pc=0x4530d2
github.com/gagliardetto/solana-go.(*Message).UnmarshalLegacy(0xc00079b738, 0xc00929df50)
    /go/pkg/mod/github.com/gagliardetto/solana-go@v1.6.1-0.20221018174950-475b9d64e462/message.go:440 +0x52c fp=0xc000886cc0 sp=0xc000886b98 pc=0xac396c
github.com/gagliardetto/solana-go.(*Message).UnmarshalWithDecoder(0xc00079b738, 0xc000886d90?)
    /go/pkg/mod/github.com/gagliardetto/solana-go@v1.6.1-0.20221018174950-475b9d64e462/message.go:310 +0xe5 fp=0xc000886d10 sp=0xc000886cc0 pc=0xac27e5
github.com/gagliardetto/solana-go.(*Transaction).UnmarshalWithDecoder(0xc00079b720, 0xc00929df50)
    /go/pkg/mod/github.com/gagliardetto/solana-go@v1.6.1-0.20221018174950-475b9d64e462/transaction.go:379 +0x245 fp=0xc000886da0 sp=0xc000886d10 pc=0xac9f25
github.com/gagliardetto/solana-go/rpc.TransactionWithMeta.GetTransaction({0xc007e294f0?, 0x0?})
    /go/pkg/mod/github.com/gagliardetto/solana-go@v1.6.1-0.20221018174950-475b9d64e462/rpc/types.go:113 +0x65 fp=0xc000886de0 sp=0xc000886da0 pc=0xb62b45

Because of the bad input, Go tried to allocate 0x163e65afc58 ~= 1423 GB worth of slices. I think we need checks to the ensure transaction truly requires that much allocation (which isn't valid in Solana anyway).

gagliardetto / solana-go

fatal error: runtime: out of memory when trying to deserialize bad transactions #120