search

open-compass / DevBench

A Comprehensive Benchmark for Software Development.
Apache License 2.0
80 stars 4 forks source link

The tutorial seems incomplete, how to evaluate it? #4

Open kartikzheng opened 1 month ago

kartikzheng commented 1 month ago

hi，according to the tutorial, execute the following command, no error, freeze here. What should I do in the next step? If you can improve this tutorial, thank you. python run.py --config Implementation --input_path ../../benchmark_data/python/TextCNN/ --model deepseek-coder-6.7b --model_source open_source --review execution --evaluate 图片1

The configuration file agent_system/baseline/open_source_config.json is as follows: { "codellama-7b-instruct": "", "codellama-13b-instruct": "", "codellama-34b-instruct": "", "deepseek-coder-1.3b-instruct": "", "deepseek-coder-6.7b-instruct": "http://XXXX:7001/v1", "deepseek-coder-33b-instruct": "", "customized-model": {"deepseek-coder-6.7b-instruct": "model_ip_address"} }

libowen2121 commented 1 month ago

Hey @kartikzheng! Thanks for raising the issue! @wenhan-wu-stephen will come to have a look!

wenhan-wu-stephen commented 1 month ago

Hi, Kartik, we have also encountered similar problems. We found that there is an error on the machine that deploys the model. Can you provide the error information on the machine? @kartikzheng