francoispqt / gojay

high performance JSON encoder/decoder with stream API for Golang
MIT License
2.11k stars 112 forks source link

Periodically flush the `buf` to the `Writer`. #78

Open abourget opened 5 years ago

abourget commented 5 years ago

Hi and thanks for the lib.

My goal is to start pushing JSON to an io.Writer while it's being JIT encoded. I noticed that EncodeObject calls enc.Write() and manages the error, that MarshalJSONObject doesn't return an error and that when calling enc.Write() myself (from within a MarshalJSONObject() func), there's no place to bubble up the error.

Surely I'm doing something a bit off here.. but I simply want to flush the buf between rows (say I have 200,000 such rows, which are objects). I also wonder how the buf would handle the getPreviousRune if I were to Write() before my Array is closed.

any tips would help, thanks a lot!

abourget commented 5 years ago

What I'm currently doing:

    enc.AddArrayKey("rows", gojay.EncodeArrayFunc(func(enc *gojay.Encoder) {
        lastIdx := len(r.Rows) - 1
        for idx, row := range r.Rows {
            enc.EncodeObject(row) // this flushes objects in between..
            if idx != lastIdx {
                enc.AppendByte(',')
            }
        }
    }))

which seems to work :)