search

symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
57 stars 3 forks source link

Openrouter returns 524 when querying models #186

Closed bauersimon closed 2 weeks ago

bauersimon commented 2 weeks ago

happened twice now

2024/06/13 11:10:12 Checking provider "openrouter" for models
2024/06/13 11:11:52 ERROR: could not query models for provider "openrouter": error, status code: 524, message: invalid character '<' looking for beginning of value
panic: ERROR: could not query models for provider "openrouter": error, status code: 524, message: invalid character '<' looking for beginning of value

goroutine 1 [running]:
log.(*Logger).Panicf(0x98a81f?, {0x9a14d4?, 0xc0002a8400?}, {0xc00031f1e8?, 0x0?, 0x13?})
        /home/simon/symflower/.devenv/go@1.21.5/go/src/log/log.go:310 +0x6e
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute(0xc0002d2000, {0x7fad9c03f8f8?, 0xc0002d2000?, 0x914180?})
        /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:243 +0x1da5
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1({0x7fad9c03f8f8, 0xc0002d2000}, {0xc0002e2000, 0x0, 0xb})
        /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34 +0x8c
github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc0002c9730, {0xc00012e010?, 0xb, 0xb})
        /home/simon/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333 +0x954
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute(0xc0002998f0, {0xc00012e010, 0xb, 0xb})
        /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37 +0x247
main.main()
        /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/main.go:11 +0x65