itchyny / gojq

Pure Go implementation of jq
MIT License
3.3k stars 119 forks source link

Allocated maps are ignored #241

Open christianhadler opened 9 months ago

christianhadler commented 9 months ago

I'm observing huge performance hits when collecting values into a an object by using reduce:

gojq:

$ time gojq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null

real    0m49.581s
user    0m0.015s
sys     0m0.031s

jq:

$ time jq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null

real    0m0.156s
user    0m0.109s
sys     0m0.062s

It seems that a new map is allocated (and filled) on every iteration:

package main

import (
    "os"
    "runtime/pprof"

    "github.com/itchyny/gojq"
)

func main() {
    fd, err := os.Create("profile.pprof")
    if err != nil {
        panic(err)
    }
    defer fd.Close()

    pprof.StartCPUProfile(fd)
    defer pprof.StopCPUProfile()

    query, err := gojq.Parse(`[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})`)
    if err != nil {
        panic(err)
    }

    code, err := gojq.Compile(query)
    if err != nil {
        panic(err)
    }

    iter := code.Run(nil)
    for {
        v, ok := iter.Next()
        if !ok {
            break
        }
        if err, ok := v.(error); ok {
            panic(err)
        }
    }
}
(pprof) list gojq.updateObject
Total: 78.57s
ROUTINE ======================== github.com/itchyny/gojq.updateObject in C:\Users\chrha\go\pkg\mod\github.com\itchyny\gojq@v0.12.15-0.20240118121937-0284dbe60fe8\func.go
     1.75s     34.67s (flat, cum) 44.13% of Total
         .          .   1604:func updateObject(v map[string]any, k string, path []any, n any, a allocator) (any, error) {
         .          .   1605:   x, ok := v[k]
         .          .   1606:   if !ok && n == struct{}{} {
         .          .   1607:           return v, nil
         .          .   1608:   }
         .          .   1609:   u, err := update(x, path, n, a)
         .          .   1610:   if err != nil {
         .          .   1611:           return nil, err
         .          .   1612:   }
         .          .   1613:   if a.allocated(v) {
         .          .   1614:           v[k] = u
         .          .   1615:           return v, nil
         .          .   1616:   }
         .      830ms   1617:   w := a.makeObject(len(v) + 1)
     770ms      6.66s   1618:   for k, v := range v {
     980ms     27.17s   1619:           w[k] = v
         .          .   1620:   }
         .       10ms   1621:   w[k] = u
         .          .   1622:   return w, nil
         .          .   1623:}
         .          .   1624:
         .          .   1625:func updateArrayIndex(v []any, i int, path []any, n any, a allocator) (any, error) {
         .          .   1626:   var x any