Open christianhadler opened 9 months ago
I'm observing huge performance hits when collecting values into a an object by using reduce:
reduce
gojq:
$ time gojq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null real 0m49.581s user 0m0.015s sys 0m0.031s
jq:
$ time jq -n '[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})' > /dev/null real 0m0.156s user 0m0.109s sys 0m0.062s
It seems that a new map is allocated (and filled) on every iteration:
package main import ( "os" "runtime/pprof" "github.com/itchyny/gojq" ) func main() { fd, err := os.Create("profile.pprof") if err != nil { panic(err) } defer fd.Close() pprof.StartCPUProfile(fd) defer pprof.StopCPUProfile() query, err := gojq.Parse(`[range(20000)] | reduce .[] as $item ({}; .[($item | tostring)] = {})`) if err != nil { panic(err) } code, err := gojq.Compile(query) if err != nil { panic(err) } iter := code.Run(nil) for { v, ok := iter.Next() if !ok { break } if err, ok := v.(error); ok { panic(err) } } }
(pprof) list gojq.updateObject Total: 78.57s ROUTINE ======================== github.com/itchyny/gojq.updateObject in C:\Users\chrha\go\pkg\mod\github.com\itchyny\gojq@v0.12.15-0.20240118121937-0284dbe60fe8\func.go 1.75s 34.67s (flat, cum) 44.13% of Total . . 1604:func updateObject(v map[string]any, k string, path []any, n any, a allocator) (any, error) { . . 1605: x, ok := v[k] . . 1606: if !ok && n == struct{}{} { . . 1607: return v, nil . . 1608: } . . 1609: u, err := update(x, path, n, a) . . 1610: if err != nil { . . 1611: return nil, err . . 1612: } . . 1613: if a.allocated(v) { . . 1614: v[k] = u . . 1615: return v, nil . . 1616: } . 830ms 1617: w := a.makeObject(len(v) + 1) 770ms 6.66s 1618: for k, v := range v { 980ms 27.17s 1619: w[k] = v . . 1620: } . 10ms 1621: w[k] = u . . 1622: return w, nil . . 1623:} . . 1624: . . 1625:func updateArrayIndex(v []any, i int, path []any, n any, a allocator) (any, error) { . . 1626: var x any
I'm observing huge performance hits when collecting values into a an object by using
reduce
:gojq:
jq:
It seems that a new map is allocated (and filled) on every iteration: