Extract model names, to obtain a human-readable name for each model

[ruiAzevedo19]

Part of #206

[ruiAzevedo19]

Based on https://github.com/symflower/eval-dev-quality/pull/216

[bauersimon]

Please try this out with the cheapest model from openrouter and post the CSV here to see how it looks like.

[ruiAzevedo19]

@bauersimon Results

OpenRouter

Command

eval-dev-quality evaluate --runs 1 --repository golang/plain --model openrouter/meta-llama/llama-3-8b-instruct

evaluation.csv

``` model-id,model-name,cost,language,repository,task,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code openrouter/meta-llama/llama-3-8b-instruct,Meta: Llama 3 8B Instruct,0.00000014,golang,golang/plain,write-tests,3,0,0,163,1302,177,1,1,1 ```

golang-summed.csv

``` model-id,model-name,cost,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code openrouter/meta-llama/llama-3-8b-instruct,Meta: Llama 3 8B Instruct,0.00000014,3,0,0,163,1302,177,1,1,1 ```

models-summed.csv

``` model-id,model-name,cost,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code openrouter/meta-llama/llama-3-8b-instruct,Meta: Llama 3 8B Instruct,0.00000014,3,0,0,163,1302,177,1,1,1 ```

Ollama

Command

eval-dev-quality evaluate --runs 1 --repository golang/plain --model ollama/qwen:0.5b

evaluation.csv

``` model-id,model-name,cost,language,repository,task,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code ollama/qwen:0.5b,qwen:0.5b,0,golang,golang/plain,write-tests,2,0,0,532,8674,810,1,0,1 ```

golang-summed.csv

``` model-id,model-name,cost,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code ollama/qwen:0.5b,qwen:0.5b,0,2,0,0,532,8674,810,1,0,1 ```

models-summed.csv

``` model-id,model-name,cost,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code ollama/qwen:0.5b,qwen:0.5b,0,2,0,0,532,8674,810,1,0,1 ```

OpenAi

Command

eval-dev-quality evaluate --runs 1 --repository golang/plain --urls custom-ollama:http://127.0.0.1:11434/v1 --model custom-ollama/qwen:0.5b

evaluation.csv

``` model-id,model-name,cost,language,repository,task,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code custom-ollama/qwen:0.5b,qwen:0.5b,0,golang,golang/plain,write-tests,2,0,0,134,3837,456,1,0,1 ```

golang-summed.csv

``` model-id,model-name,cost,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code custom-ollama/qwen:0.5b,qwen:0.5b,0,2,0,0,134,3837,456,1,0,1 ```

models-summed.csv

``` model-id,model-name,cost,score,coverage,files-executed,generate-tests-for-file-character-count,processing-time,response-character-count,response-no-error,response-no-excess,response-with-code custom-ollama/qwen:0.5b,qwen:0.5b,0,2,0,0,134,3837,456,1,0,1 ```

[bauersimon]

Please still always put the model id first.

[bauersimon]

Just asking myself if we should put the model names in quotes in the csv... As long as no name contains a , it should be good.

[bauersimon]

Just asking myself if we should put the model names in quotes in the csv... As long as no name contains a , it should be good.

Not necessary: https://pkg.go.dev/encoding/csv#Writer.Write. Will quote as required.