Parsing of benchmark results with panic/timeout causes another panic

radeksimko commented 2 years ago

I'm not sure what the best course of action is, but panic doesn't seem right in any case:

go test ./internal/langserver/handlers \
    -bench=InitializeFolder_basic \
    -run=^# \
    -benchtime=60s \
    -timeout=30m | tee /home/runner/work/_temp/benchmarks.txt

goos: linux
goarch: amd64
pkg: github.com/hashicorp/terraform-ls/internal/langserver/handlers
cpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
BenchmarkInitializeFolder_basic/local-single-module-no-provider-2               2354      29887759 ns/op     1466327 B/op      23874 allocs/op
BenchmarkInitializeFolder_basic/local-single-submodule-no-provider-2             471     147021517 ns/op     2709140 B/op      40481 allocs/op
BenchmarkInitializeFolder_basic/local-single-module-random-2                     510     143119210 ns/op     1911226 B/op      27656 allocs/op
BenchmarkInitializeFolder_basic/local-single-module-aws-2                         51    1371407234 ns/op    105128302 B/op    745133 allocs/op
BenchmarkInitializeFolder_basic/aws-consul-2                                      42    1447751904 ns/op    127841282 B/op   1055052 allocs/op
BenchmarkInitializeFolder_basic/aws-eks-2                                         28    2296038135 ns/op    199662055 B/op   1481714 allocs/op
BenchmarkInitializeFolder_basic/aws-vpc-2                                         46    1530870496 ns/op    123258121 B/op    862732 allocs/op
BenchmarkInitializeFolder_basic/google-project-2                                  38    2780695563 ns/op    148429337 B/op   1076845 allocs/op
BenchmarkInitializeFolder_basic/google-network-2                                  28    2585281011 ns/op    133152685 B/op    878487 allocs/op
BenchmarkInitializeFolder_basic/google-gke-2                                      15    4587726688 ns/op    131472710 B/op    844753 allocs/op
BenchmarkInitializeFolder_basic/k8s-metrics-server-2                              56    1301099738 ns/op    60277646 B/op     378735 allocs/op
BenchmarkInitializeFolder_basic/k8s-dashboard-2                             SIGQUIT: quit
PC=0x46ae5c m=0 sigcode=0

goroutine 181042 [running]:
runtime.memclrNoHeapPointers()

See whole log at https://github.com/hashicorp/terraform-ls/runs/6137664889?check_suite_focus=true#step:7:10

  BENCHDATA="go run go.bobheadxi.dev/gobenchdata"

  BENCH_PATH="/home/runner/work/_temp/benchmarks.txt"
  DATA_PATH="/home/runner/work/_temp/benchdata.json"
  RESULTS_PATH="/home/runner/work/_temp/benchdata-results.json"
  CHECKS_CONFIG_PATH="/home/runner/work/terraform-ls/terraform-ls/.github/gobenchdata-checks.yml"

  cat $BENCH_PATH | $BENCHDATA --json ${DATA_PATH} -v "${GITHUB_SHA}" -t "ref=${GITHUB_REF}"

  $BENCHDATA checks eval \
    ${DATA_PATH} \
    ${DATA_PATH} \
    --checks.config ${CHECKS_CONFIG_PATH}

panic: BenchmarkInitializeFolder_basic/k8s-dashboard-2: could not parse run: strconv.Atoi: parsing "SIGQUIT: quit": invalid syntax (line: BenchmarkInitializeFolder_basic/k8s-dashboard-2                               SIGQUIT: quit)

goroutine 1 [running]:
main.main()
    /home/runner/go/pkg/mod/go.bobheadxi.dev/gobenchdata@v1.0.2/main.go:230 +0x135d
exit status 2

bobheadxi commented 2 years ago

Looks like I just panic on every error in main() - https://github.com/bobheadxi/gobenchdata/pull/52 replaces those with print + exit, which hopefully looks less alarming than the panic output!

bobheadxi commented 2 years ago

I've published https://github.com/bobheadxi/gobenchdata/releases/tag/v1.0.3 - let me know if you'd like some other behaviour in this scenario!

bobheadxi / gobenchdata

Parsing of benchmark results with panic/timeout causes another panic #51