search

symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
137 stars 5 forks source link

Docker runtime broken on main #302

Closed bauersimon closed 3 months ago

bauersimon commented 4 months ago 
make install && eval-dev-quality evaluate --model symflower/symbolic-execution --repository golang/plain --runtime docker
2024/07/31 15:55:49 Writing results to evaluation-2024-07-31-15:55:49
2024/07/31 15:55:49 Spawning new log file at evaluation-2024-07-31-15:55:49/evaluation.log
2024/07/31 15:55:49 $ docker volume create evaluation-volume
2024/07/31 15:55:49 evaluation-volume
2024/07/31 15:55:49 $ docker run -e PROVIDER_TOKEN -v evaluation-volume:/app/evaluation --rm ghcr.io/symflower/eval-dev-quality:main eval-dev-quality evaluate --model symflower/symbolic-execution --result-path /app/evaluation/symflower_symbolic-execution --repository golang/plain
2024/07/31 15:55:49 2024/07/31 13:55:49 Writing results to /app/evaluation/symflower_symbolic-execution
2024/07/31 13:55:49 ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied
2024/07/31 15:55:49 panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied [recovered]
    panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied

goroutine 1 [running]:
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Initialize.func1()
    /app/cmd/eval-dev-quality/cmd/evaluate.go:117 +0x45
panic({0x8d6e40?, 0xc00028a7d0?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
log.(*Logger).Panicf(0xc000286690, {0x9731a2?, 0x3b?}, {0xc00021fc10?, 0xc0002900c0?, 0x13?})
    /usr/local/go/src/log/log.go:310 +0x6e
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Initialize(0xc0002b6000, {0x0?, 0x2?, 0x0?})
    /app/cmd/eval-dev-quality/cmd/evaluate.go:215 +0xa6c
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute(0xc0002b6000, {0xc0002a7110, 0x0, 0x7})
    /app/cmd/eval-dev-quality/cmd/evaluate.go:402 +0x9c
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1({0x7f601f95e0f8, 0xc0002b6000}, {0xc0002a7110, 0x0, 0x7})
    /app/cmd/eval-dev-quality/cmd/command.go:38 +0x12d
github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc0002a62a0, {0xc000022090, 0x7, 0x7})
    /go/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333 +0xaf9
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute(0xc000286690, {0xc000022090, 0x7, 0x7})
    /app/cmd/eval-dev-quality/cmd/command.go:41 +0x25c
main.main()
    /app/cmd/eval-dev-quality/main.go:11 +0x65
2024/07/31 15:55:50 ERROR: 2024/07/31 13:55:49 Writing results to /app/evaluation/symflower_symbolic-execution
2024/07/31 13:55:49 ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied
panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied [recovered]
    panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied

goroutine 1 [running]:
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Initialize.func1()
    /app/cmd/eval-dev-quality/cmd/evaluate.go:117 +0x45
panic({0x8d6e40?, 0xc00028a7d0?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
log.(*Logger).Panicf(0xc000286690, {0x9731a2?, 0x3b?}, {0xc00021fc10?, 0xc0002900c0?, 0x13?})
    /usr/local/go/src/log/log.go:310 +0x6e
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Initialize(0xc0002b6000, {0x0?, 0x2?, 0x0?})
    /app/cmd/eval-dev-quality/cmd/evaluate.go:215 +0xa6c
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute(0xc0002b6000, {0xc0002a7110, 0x0, 0x7})
    /app/cmd/eval-dev-quality/cmd/evaluate.go:402 +0x9c
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1({0x7f601f95e0f8, 0xc0002b6000}, {0xc0002a7110, 0x0, 0x7})
    /app/cmd/eval-dev-quality/cmd/command.go:38 +0x12d
github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc0002a62a0, {0xc000022090, 0x7, 0x7})
    /go/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333 +0xaf9
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute(0xc000286690, {0xc000022090, 0x7, 0x7})
    /app/cmd/eval-dev-quality/cmd/command.go:41 +0x25c
main.main()
    /app/cmd/eval-dev-quality/main.go:11 +0x65
: 2024/07/31 13:55:49 Writing results to /app/evaluation/symflower_symbolic-execution
2024/07/31 13:55:49 ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied
panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied [recovered]
    panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied

goroutine 1 [running]:
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Initialize.func1()
    /app/cmd/eval-dev-quality/cmd/evaluate.go:117 +0x45
panic({0x8d6e40?, 0xc00028a7d0?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
log.(*Logger).Panicf(0xc000286690, {0x9731a2?, 0x3b?}, {0xc00021fc10?, 0xc0002900c0?, 0x13?})
    /usr/local/go/src/log/log.go:310 +0x6e
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Initialize(0xc0002b6000, {0x0?, 0x2?, 0x0?})
    /app/cmd/eval-dev-quality/cmd/evaluate.go:215 +0xa6c
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute(0xc0002b6000, {0xc0002a7110, 0x0, 0x7})
    /app/cmd/eval-dev-quality/cmd/evaluate.go:402 +0x9c
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1({0x7f601f95e0f8, 0xc0002b6000}, {0xc0002a7110, 0x0, 0x7})
    /app/cmd/eval-dev-quality/cmd/command.go:38 +0x12d
github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc0002a62a0, {0xc000022090, 0x7, 0x7})
    /go/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333 +0xaf9
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute(0xc000286690, {0xc000022090, 0x7, 0x7})
    /app/cmd/eval-dev-quality/cmd/command.go:41 +0x25c
main.main()
    /app/cmd/eval-dev-quality/main.go:11 +0x65
: exit status 2
2024/07/31 15:55:50 $ docker run -d --name volume-fetch -v evaluation-volume:/data busybox true
2024/07/31 15:55:50 6da845a3d21cf6e5f8560e2f65275129e9745f8e5cf8b9eac7dd1254c484a588
2024/07/31 15:55:50 $ docker cp volume-fetch:/data/. evaluation-2024-07-31-15:55:49
2024/07/31 15:55:50 copying between containers is not supported
2024/07/31 15:55:50 $ docker rm volume-fetch
2024/07/31 15:55:50 volume-fetch
2024/07/31 15:55:50 $ docker volume rm evaluation-volume
2024/07/31 15:55:50 evaluation-volume
copying between containers is not supported
: copying between containers is not supported
: exit status 1
2024/07/31 15:55:50 Could not parse arguments: exit status 1
copying between containers is not supported

github.com/symflower/eval-dev-quality/util.CommandWithResult
    /home/simon/repos/eval-dev-quality/util/exec.go:60
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).evaluateDocker
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:653
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:483
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:40
github.com/jessevdk/go-flags.(*Parser).ParseArgs
    /home/simon/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:43
main.main
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).evaluateDocker
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:662
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:483
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:40
github.com/jessevdk/go-flags.(*Parser).ParseArgs
    /home/simon/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:43
main.main
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
copying between containers is not supported

panic: Could not parse arguments: exit status 1
copying between containers is not supported

github.com/symflower/eval-dev-quality/util.CommandWithResult
    /home/simon/repos/eval-dev-quality/util/exec.go:60
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).evaluateDocker
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:653
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:483
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:40
github.com/jessevdk/go-flags.(*Parser).ParseArgs
    /home/simon/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:43
main.main
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).evaluateDocker
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:662
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:483
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:40
github.com/jessevdk/go-flags.(*Parser).ParseArgs
    /home/simon/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:43
main.main
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
    /home/simon/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
copying between containers is not supported

goroutine 1 [running]:
github.com/symflower/eval-dev-quality/log.(*Logger).Panicf(0xc000130050, {0x9c5745?, 0x7?}, {0xc00021fed8?, 0x0?, 0x960400?})
    /home/simon/repos/eval-dev-quality/log/logger.go:124 +0x94
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute(0xc000130050, {0xc000024090?, 0x7, 0x7})
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:48 +0x2f2
main.main()
    /home/simon/repos/eval-dev-quality/cmd/eval-dev-quality/main.go:11 +0x17c
bauersimon commented 4 months ago

Not sure what I did wrong. I had one semi-working run but then it encountered the copying between containers is not supported error that is below in the logs...

docker volume list and docker ps is empty though.

bauersimon commented 4 months ago

And now every time I run it won't even create the log directory...

Munsio commented 4 months ago

It has to be something different because of: 

ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied
2024/07/31 15:55:49 panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied [recovered]
    panic: ERROR: mkdir /app/evaluation/symflower_symbolic-execution: permission denied
Munsio commented 4 months ago

@bauersimon could you run docker pull ghcr.io/symflower/eval-dev-quality:main and try again pls.

bauersimon commented 3 months ago

Working again :confetti_ball: