search

ugorji / go

idiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]
MIT License
1.86k stars 295 forks source link

codecgen error: error running 'go run codecgen-main-5831.generated.go' ... #276

Closed afiskon closed 5 years ago

afiskon commented 5 years ago

Hello,

We would like to use codecgen in our project https://github.com/insolar/insolar However we faced some problems. Hopefully you could help.

The environment is:

$ go version
go version go1.11.2 darwin/amd64

The example project currently uses serialisation based on reflection. File types/types.go:

package types

type Class int
const (
    WARIOR Class = iota
    MAGE   Class = iota
)

type Spell int
const (
    FIREBALL Spell = iota
    THUNDERBOLT Spell = iota
)

type Weapon int
const (
    SWORD Weapon = iota
    BOW Weapon = iota
)

type WariorInfo struct {
    Weapon Weapon
    ArrowsNumber int
}

type MageInfo struct {
    Spellbook []Spell
    Mana int
}

type Hero struct {
    Name string
    HP int
    XP int
    WariorInfo *WariorInfo
    MageInfo *MageInfo
}

File main.go:

package main

import (
    . "go-sandbox/codec-examples-gen/types"
    "github.com/ugorji/go/codec"
    "log"
)

func main() {
    var (
        cborHandle codec.CborHandle
        err error
    )

    //v1 := Hero{ "Alex", 123, 456, &WariorInfo{ BOW, 10 }, nil}
    v1 := Hero{ "Bob", 234, 567, nil, &MageInfo{ []Spell{FIREBALL, THUNDERBOLT}, 42 }}

    var bs []byte
    enc := codec.NewEncoderBytes(&bs, &cborHandle)
    err = enc.Encode(v1)
    if err != nil {
        log.Fatalf("enc.Encode() failed: %v", err)
    }
    log.Printf("bs = %X, len(bs) = %d, cap(bs) = %d", bs, len(bs), cap(bs))

    // Decode bs to v2

    var v2 Hero
    dec := codec.NewDecoderBytes(bs, &cborHandle)
    err = dec.Decode(&v2)
    if err != nil {
        log.Fatalf("dec.Decode() failed: %v", err)
    }

    log.Printf("v2 = %v", v2)
    if v2.WariorInfo != nil{
        log.Printf("WariorInfo = %v", *v2.WariorInfo)
    }
    if v2.MageInfo != nil {
        log.Printf("MageInfo = %v", *v2.MageInfo)
    }
}

codec and codecgen were downloaded using go get:

go get -u github.com/ugorji/go/codec
go get -u github.com/ugorji/go/codec/codecgen

Now we would like to generate the code using types/types.go as an input file:

codecgen -o types/types.gen.go types/types.go

The utility fails with the following error:

codecgen error: error running 'go run codecgen-main-5831.generated.go': exit status 2, console: # command-line-arguments
./codecgen-main-5831.generated.go:8:2: undefined: types.CodecGenTempWrite5831

We believe we did everything according to the manual and that this is a bug.

If you need any additional information (an output of strace maybe?) we would be happy to provide it.

ugorji commented 5 years ago

Please run codecgen from the types directory, and let me know how that works.

I'm currently away from my computer, so trying to remember from memory.

afiskon commented 5 years ago

Thank you for a quick reply. It seems to work from types/ directory, at least the file is successfully created. It's pretty late in my timezone though, thus I will be able to properly check it only tomorrow.

afiskon commented 5 years ago

OK, the code was generated, however it doesn't seem to work very correctly.

I tried two programs. The first one is based on io.Writer interface:

package main

import (
    "bytes"
    "github.com/ugorji/go/codec"
    . "go-sandbox/codec-examples-gen/types"
    "log"
)

func main() {
    var cborHandle codec.CborHandle

    //v1 := Hero{ "Alex", 123, 456, &WariorInfo{ BOW, 10 }, nil}
    v1 := Hero{ "Bob", 234, 567, nil, &MageInfo{ []Spell{FIREBALL, THUNDERBOLT}, 42 }}

    var buff bytes.Buffer
    buff.Grow(4096)
    enc := codec.NewEncoder(&buff, &cborHandle)
    v1.CodecEncodeSelf(enc)
    bytes := buff.Bytes()
    log.Printf("bs = %X, len(bs) = %d, cap(bs) = %d", bytes, len(bytes), cap(bytes))

    // Decode buff to v2

    var v2 Hero
    dec := codec.NewDecoderBytes(bytes, &cborHandle)
    v2.CodecDecodeSelf(dec)

    log.Printf("v2 = %v", v2)
    if v2.WariorInfo != nil{
        log.Printf("WariorInfo = %v", *v2.WariorInfo)
    }
    if v2.MageInfo != nil {
        log.Printf("MageInfo = %v", *v2.MageInfo)
    }
}

It fails with the following error:

$ ./codec-examples-gen 
2018/12/06 16:24:43 bs = , len(bs) = 0, cap(bs) = 4096
panic: EOF

goroutine 1 [running]:
github.com/ugorji/go/codec.(*bytesDecReader).readn1(...)
    /Users/eax/go/src/github.com/ugorji/go/codec/decode.go:1007
github.com/ugorji/go/codec.(*decReaderSwitch).readn1(0xc000174050, 0x0)
    /Users/eax/go/src/github.com/ugorji/go/codec/decode.go:2309 +0xa6
github.com/ugorji/go/codec.(*cborDecDriver).readNextBd(0xc00013e160)
    /Users/eax/go/src/github.com/ugorji/go/codec/cbor.go:313 +0x2f
github.com/ugorji/go/codec.(*cborDecDriver).ContainerType(0xc00013e160, 0xc000174000)
    /Users/eax/go/src/github.com/ugorji/go/codec/cbor.go:326 +0xcf
go-sandbox/codec-examples-gen/types.(*Hero).CodecDecodeSelf(0xc000134360, 0xc000174000)
    /Users/eax/go/src/go-sandbox/codec-examples-gen/types/types.gen.go:666 +0xb1
main.main()
    /Users/eax/go/src/go-sandbox/codec-examples-gen/main.go:27 +0x360

Here is the second program, which uses []byte:

package main

import (
    "github.com/ugorji/go/codec"
    . "go-sandbox/codec-examples-gen/types"
    "log"
)

func main() {
    var cborHandle codec.CborHandle

    //v1 := Hero{ "Alex", 123, 456, &WariorInfo{ BOW, 10 }, nil}
    v1 := Hero{ "Bob", 234, 567, nil, &MageInfo{ []Spell{FIREBALL, THUNDERBOLT}, 42 }}

    bs := make([]byte, 182)
    enc := codec.NewEncoderBytes(&bs, &cborHandle)
    v1.CodecEncodeSelf(enc)
    log.Printf("bs = %X, len(bs) = %d, cap(bs) = %d", bs, len(bs), cap(bs))

    // Decode bs to v2

    var v2 Hero
    dec := codec.NewDecoderBytes(bs, &cborHandle)
    v2.CodecDecodeSelf(dec)

    log.Printf("v2 = %v", v2)
    if v2.WariorInfo != nil{
        log.Printf("WariorInfo = %v", *v2.WariorInfo)
    }
    if v2.MageInfo != nil {
        log.Printf("MageInfo = %v", *v2.MageInfo)
    }
}

And its output:

$ ./codec-examples-gen 
2018/12/06 16:26:02 bs = A5644E616D6563426F6262485018EA6258501902376A576172696F72496E666
FF6684D616765496E666FA2695370656C6C626F6F6B820001644D616E61182A
00000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000, len(bs) = 182,
 cap(bs) = 182
2018/12/06 16:26:02 v2 = {Bob 234 567 <nil> 0xc000096b20}
2018/12/06 16:26:02 MageInfo = {[0 1] 42}

Note that without increasing the buffer size ( bs := make([]byte, 182) ) the code fails in the similar way the first program does. Apparently, unlike the serialization based on reflection, this code is not capable to grow the buffer dynamically.

Although the second program works, currently the API doesn't allow to determine the resulting size of serialized object, and doesn't return any error indicator if the buffer was too small. Thus there is no way to guess required buffer size and truncate it after the serialization on the calling side.

I'm going to investigate how it can be fixed. However I guess as an author you might be able to find the solution much faster. (To be honest, currently I have very little experience of programming in Go.)

ugorji commented 5 years ago

The problem is that you are calling v1.CodecEncodeSelf(...). That's wrong. That is not guaranteed to flush any interim buffers, etc. That is a call that the Encoder may use during encoding, if a type can encode itself. But it is a part of the Encoding/Decoding process, not the full thing.

In your case, by calling that, the "end" doesn't happen where it all comes together, so the bytes still is empty, etc.

What you should do is the supported, documented model, i.e.:

enc := codec.NewEncoderBytes(&bs, &cborHandle)
enc.Encode(v1) or enc.MustEncode(v1)

dec := codec.NewDecoderBytes(bs, &cborHandle)
dec.Decode(v2) or dec.MustDecode(v2)
ugorji commented 5 years ago

Also, do some metrics when deciding to use codecgen.

codecgen limitations are that it doesn't honor MissingFielder interface or PreferArrayOverSlice option (which are infrequently used), and it requires a (manual) code-generation stage whenever types are changed.

Currently, codecgen gives about:

Hope this helps.

Latest benchcmp results:

benchmark                                                                       old ns/op     new ns/op     delta
BenchmarkCodecQuickSuite/cbor-bd1......../Benchmark__Cbor_______Encode-8        39112         28111         -28.13%
BenchmarkCodecQuickSuite/cbor-bd1-buf1024/Benchmark__Cbor_______Encode-8        45448         31452         -30.80%
BenchmarkCodecQuickSuite/cbor-bd1-io...../Benchmark__Cbor_______Encode-8        42906         32162         -25.04%
BenchmarkCodecQuickSuite/cbor-bd1........#01/Benchmark__Cbor_______Decode-8     89815         57057         -36.47%
BenchmarkCodecQuickSuite/cbor-bd1-buf1024#01/Benchmark__Cbor_______Decode-8     112089        77668         -30.71%
BenchmarkCodecQuickSuite/cbor-bd1-io.....#01/Benchmark__Cbor_______Decode-8     137342        95314         -30.60%
BenchmarkCodecQuickSuite/json-bd1......../Benchmark__Json_______Encode-8        130866        112940        -13.70%
BenchmarkCodecQuickSuite/json-bd1-buf1024/Benchmark__Json_______Encode-8        137318        124362        -9.44%
BenchmarkCodecQuickSuite/json-bd1-io...../Benchmark__Json_______Encode-8        137550        124890        -9.20%
BenchmarkCodecQuickSuite/json-bd1........#01/Benchmark__Json_______Decode-8     194751        149503        -23.23%
BenchmarkCodecQuickSuite/json-bd1-buf1024#01/Benchmark__Json_______Decode-8     222511        177808        -20.09%
BenchmarkCodecQuickSuite/json-bd1-io.....#01/Benchmark__Json_______Decode-8     382285        327606        -14.30%

benchmark                                                                       old allocs     new allocs     delta
BenchmarkCodecQuickSuite/cbor-bd1......../Benchmark__Cbor_______Encode-8        22             8              -63.64%
BenchmarkCodecQuickSuite/cbor-bd1-buf1024/Benchmark__Cbor_______Encode-8        24             10             -58.33%
BenchmarkCodecQuickSuite/cbor-bd1-io...../Benchmark__Cbor_______Encode-8        24             10             -58.33%
BenchmarkCodecQuickSuite/cbor-bd1........#01/Benchmark__Cbor_______Decode-8     368            329            -10.60%
BenchmarkCodecQuickSuite/cbor-bd1-buf1024#01/Benchmark__Cbor_______Decode-8     396            359            -9.34%
BenchmarkCodecQuickSuite/cbor-bd1-io.....#01/Benchmark__Cbor_______Decode-8     394            355            -9.90%
BenchmarkCodecQuickSuite/json-bd1......../Benchmark__Json_______Encode-8        22             8              -63.64%
BenchmarkCodecQuickSuite/json-bd1-buf1024/Benchmark__Json_______Encode-8        24             10             -58.33%
BenchmarkCodecQuickSuite/json-bd1-io...../Benchmark__Json_______Encode-8        24             10             -58.33%
BenchmarkCodecQuickSuite/json-bd1........#01/Benchmark__Json_______Decode-8     453            409            -9.71%
BenchmarkCodecQuickSuite/json-bd1-buf1024#01/Benchmark__Json_______Decode-8     459            419            -8.71%
BenchmarkCodecQuickSuite/json-bd1-io.....#01/Benchmark__Json_______Decode-8     543            499            -8.10%

benchmark                                                                       old bytes     new bytes     delta
BenchmarkCodecQuickSuite/cbor-bd1......../Benchmark__Cbor_______Encode-8        2208          1664          -24.64%
BenchmarkCodecQuickSuite/cbor-bd1-buf1024/Benchmark__Cbor_______Encode-8        2352          1808          -23.13%
BenchmarkCodecQuickSuite/cbor-bd1-io...../Benchmark__Cbor_______Encode-8        2353          1808          -23.16%
BenchmarkCodecQuickSuite/cbor-bd1........#01/Benchmark__Cbor_______Decode-8     35456         34264         -3.36%
BenchmarkCodecQuickSuite/cbor-bd1-buf1024#01/Benchmark__Cbor_______Decode-8     46301         45109         -2.57%
BenchmarkCodecQuickSuite/cbor-bd1-io.....#01/Benchmark__Cbor_______Decode-8     46256         45064         -2.58%
BenchmarkCodecQuickSuite/json-bd1......../Benchmark__Json_______Encode-8        2368          1824          -22.97%
BenchmarkCodecQuickSuite/json-bd1-buf1024/Benchmark__Json_______Encode-8        2512          1969          -21.62%
BenchmarkCodecQuickSuite/json-bd1-io...../Benchmark__Json_______Encode-8        2512          1969          -21.62%
BenchmarkCodecQuickSuite/json-bd1........#01/Benchmark__Json_______Decode-8     49176         47824         -2.75%
BenchmarkCodecQuickSuite/json-bd1-buf1024#01/Benchmark__Json_______Decode-8     49631         49630         -0.00%
BenchmarkCodecQuickSuite/json-bd1-io.....#01/Benchmark__Json_______Decode-8     61640         60288         -2.19%
ugorji commented 5 years ago

See https://godoc.org/github.com/ugorji/go/codec for API docs also.

afiskon commented 5 years ago

OK, thank you a lot! Now everything works and I see ~30% speed up on a simple encode+decode benchmark. I'm closing this issue since the original bug is very minor one and go generate ./types works just fine anyway. Once again, thank you a lot!