search

cue-lang / cue

The home of the CUE language! Validate and define text-based and dynamic configuration
https://cuelang.org
Apache License 2.0
4.92k stars 278 forks source link

evaluator: performance regression on 0.6.0 #2559

Open MarcWort opened 11 months ago

MarcWort commented 11 months ago

What version of CUE are you using (cue version)?

➜  cue git:(33b7393) ✗ ./cue-33b7393 version
cue version v0.0.0-20230808110549-33b73930b8fc

go version go1.21.0
      -buildmode exe
       -compiler gc
  DefaultGODEBUG panicnil=1
     CGO_ENABLED 1
          GOARCH amd64
            GOOS linux
         GOAMD64 v1
             vcs git
    vcs.revision 33b73930b8fc8a719a88978be0b7e019ba34e247
        vcs.time 2023-08-08T11:05:49Z
    vcs.modified false
➜  cue git:(97d7109d) ✗ ./cue-97d7109 version
cue version v0.0.0-20230809090532-97d7109df4d0

go version go1.21.0
      -buildmode exe
       -compiler gc
  DefaultGODEBUG panicnil=1
     CGO_ENABLED 1
          GOARCH amd64
            GOOS linux
         GOAMD64 v1
             vcs git
    vcs.revision 97d7109df4d03e7ecc77daa12dbc8b6a247e4d37
        vcs.time 2023-08-09T09:05:32Z
    vcs.modified true

Does this issue reproduce with the latest stable release?

Yes also running on 0.6.0.

What did you do?

I noticed that when running cue 0.6.0 that it was approximately 10x slower on my rules. Bisecting led me to commit 97d7109df4d03e7ecc77daa12dbc8b6a247e4d37 where the performance regression started but only when using a "_tool.cue" file.

➜  example-cue-gh git:(main) hyperfine "../cue/cue-33b7393 build-rules" && hyperfine "../cue/cue-97d7109 build-rules"
Benchmark 1: ../cue/cue-33b7393 build-rules
  Time (mean ± σ):     282.4 ms ±  15.0 ms    [User: 436.7 ms, System: 33.0 ms]
  Range (min … max):   266.5 ms … 308.8 ms    10 runs

Benchmark 1: ../cue/cue-97d7109 build-rules
  Time (mean ± σ):      5.881 s ±  0.450 s    [User: 8.591 s, System: 0.117 s]
  Range (min … max):    5.337 s …  6.769 s    10 runs

but not when directly using cue export

➜  example-cue-gh git:(main) hyperfine "../cue/cue-33b7393 export rules/rule.cue --out yaml" && hyperfine "../cue/cue-97d7109 export rules/rule.cue --out yaml"
Benchmark 1: ../cue/cue-33b7393 export rules/rule.cue --out yaml
  Time (mean ± σ):     157.8 ms ±   7.6 ms    [User: 241.0 ms, System: 21.8 ms]
  Range (min … max):   142.5 ms … 176.3 ms    18 runs

Benchmark 1: ../cue/cue-97d7109 export rules/rule.cue --out yaml
  Time (mean ± σ):     160.7 ms ±  11.3 ms    [User: 252.4 ms, System: 15.9 ms]
  Range (min … max):   147.7 ms … 189.6 ms    19 runs

You can reproduce my tests with my example repo: https://github.com/MarcWort/example-cue

What did you expect to see?

Not such a strong performance impact. My actual project now takes 4 minutes instead of 30secs. I tried some changes on the ruleset but havent found any particular section where the performance hit appears.

mvdan commented 11 months ago

What about CUE versions f6e49bdebce2873078d8ea43a05b6c1ec0d290af (when Recurse was first added) and 0554d4e559e12914b9f5802a416c55a307517e06 (the commit right before)?

https://github.com/cue-lang/cue/commit/97d7109df4d03e7ecc77daa12dbc8b6a247e4d37 was only intended to get back the original behavior, as Marcel's change to add Recurse was just meant to be a refactor, but it introduced some regressions.

MarcWort commented 11 months ago

0554d4e559e12914b9f5802a416c55a307517e06 is a bit slower but no as bad as 97d7109df4d03e7ecc77daa12dbc8b6a247e4d37

➜  example-cue-gh git:(main) hyperfine "../cue/cue-33b7393 build-rules" && hyperfine "../cue/cue-97d7109 build-rules" && hyperfine "../cue/cue-f6e49bd build-rules" && hyperfine "../cue/cue-0554d4e build-rules"
Benchmark 1: ../cue/cue-33b7393 build-rules
  Time (mean ± σ):     238.8 ms ±   9.9 ms    [User: 382.5 ms, System: 19.5 ms]
  Range (min … max):   226.6 ms … 258.9 ms    12 runs

Benchmark 1: ../cue/cue-97d7109 build-rules
  Time (mean ± σ):      5.774 s ±  0.448 s    [User: 8.408 s, System: 0.123 s]
  Range (min … max):    5.279 s …  6.421 s    10 runs

Benchmark 1: ../cue/cue-f6e49bd build-rules
  Time (mean ± σ):     272.4 ms ±   8.1 ms    [User: 434.0 ms, System: 24.8 ms]
  Range (min … max):   261.6 ms … 284.6 ms    10 runs

Benchmark 1: ../cue/cue-0554d4e build-rules
  Time (mean ± σ):     824.0 ms ±  33.7 ms    [User: 1204.3 ms, System: 30.2 ms]
  Range (min … max):   788.9 ms … 904.5 ms    10 runs

when using export all of them are fast

➜  example-cue-gh git:(main) hyperfine "../cue/cue-33b7393 export rules/rule.cue --out yaml" && hyperfine "../cue/cue-97d7109 export rules/rule.cue --out yaml" && hyperfine "../cue/cue-f6e49bd export rules/rule.cue --out yaml" && hyperfine "../cue/cue-0554d4e export rules/rule.cue --out yaml"
Benchmark 1: ../cue/cue-33b7393 export rules/rule.cue --out yaml
  Time (mean ± σ):     133.9 ms ±   7.6 ms    [User: 207.6 ms, System: 15.2 ms]
  Range (min … max):   121.0 ms … 151.2 ms    21 runs

Benchmark 1: ../cue/cue-97d7109 export rules/rule.cue --out yaml
  Time (mean ± σ):     136.5 ms ±   6.5 ms    [User: 208.3 ms, System: 15.7 ms]
  Range (min … max):   123.1 ms … 148.9 ms    22 runs

Benchmark 1: ../cue/cue-f6e49bd export rules/rule.cue --out yaml
  Time (mean ± σ):     133.8 ms ±   6.0 ms    [User: 205.0 ms, System: 14.2 ms]
  Range (min … max):   120.8 ms … 150.8 ms    22 runs

Benchmark 1: ../cue/cue-0554d4e export rules/rule.cue --out yaml
  Time (mean ± σ):     128.0 ms ±   6.4 ms    [User: 201.6 ms, System: 12.5 ms]
  Range (min … max):   119.8 ms … 141.5 ms    23 runs
mvdan commented 10 months ago

We are currently focusing on v0.7, whose main goal is performance improvements. Marcel already has some big changes coming up in that direction, so I'm inclined to let them hit master first and then see where we stand with your benchmark.

MarcWort commented 10 months ago

Sure, let me know if there is any early dev branch to test. Im willing to help.

MarcWort commented 7 months ago

I know that 0.7.0 has no performance improvements, but just to keep you updated, here is a performance test on the current 0.7.0-rc.1 tag:

➜  example-cue-gh git:(main) hyperfine "../cue/cue-b36b477 build-rules" 
Benchmark 1: ../cue/cue-b36b477 build-rules
  Time (mean ± σ):      6.676 s ±  0.657 s    [User: 9.428 s, System: 0.120 s]
  Range (min … max):    5.711 s …  7.553 s    10 runs

➜  example-cue-gh git:(main) hyperfine "../cue/cue-97d7109 build-rules" 
Benchmark 1: ../cue/cue-97d7109 build-rules
  Time (mean ± σ):      5.715 s ±  0.417 s    [User: 8.266 s, System: 0.111 s]
  Range (min … max):    5.401 s …  6.477 s    10 runs