Unable to run benchmark tasks on windows due to incorrect directory creation syntax

[mkovelamudi]

Unable to run benchmark tasks using "eval-dev-quality evaluate" command on windows due to usage of ":" in the directory creation syntax. Check the attached screenshot of the error.

Steps to reproduce

1. Install Go, Git
2. Clone the repository and install application.
3. Run "eval-dev-quality evaluate --model=openrouter/meta-llama/llama-3-70b-instruct" command.

System Configuration Windows 10 pro

Findings I found that in evaluat.go file (line 67), following format is used for directory creation. command.ResultPath = strings.ReplaceAll(command.ResultPath, "%datetime%", evaluationTimestamp.Format("2006-01-02-15:04:05"))

Usage of ":", I believe is not supported for directory creation in windows.

[zimmski]

Thanks @mkovelamudi for the bug report. We only test on Linux but there should be no reason that this doesn't work on Windows. Let me give it a try for a few minutes and get back to you. I do not have a Windows/MacOS machine, so working on non-Linux thingies takes a bit.

[zimmski]

Added the easier OS first: MacOS support https://github.com/symflower/eval-dev-quality/pull/102 which shows that we can port to other OSes. So taking a look now in supporting Windows.

[zimmski]

@mkovelamudi will continue tomorrow. I feared that Windows is more work, and i was right. You can track the progress at https://github.com/symflower/eval-dev-quality/pull/103. Looks like the Symflower cache cannot be loaded for some reason. Need to dig in.

[mkovelamudi]

@zimmski please take your time. I was just trying out and observed the issue. I just wanted to report it.

[zimmski]

@mkovelamudi that was quite a rid of Windows-nonsense to figure out. Please give it a try and let me know how the eval runs on your machine!