search

FudanSELab / ClassEval

Benchmark ClassEval for class-level code generation.
MIT License
123 stars 8 forks source link

Where Can I Submit my Models for evaluation? #3

Closed ajinkya123-robo closed 5 months ago

ajinkya123-robo commented 7 months ago

Hello, This is a great initiative! Where can i submit my models for evals? Thanks

wkx228 commented 7 months ago

Hello, This is a great initiative! Where can i submit my models for evals? Thanks

We provide inference.py script to generate class-level code outputs. If your model is available on hugging face and not in the default model list, you can set the argument --model to 10, and set the model’s path by argument --checkpoint to run the script. You can follow the README for more details and the specific description for different arguments can be obtained by using --help. Note that we use API AutoModelForCausalLM.from_pretrained and model.generate to load and run most models, so if your model is unavailable on hugging face, you might need to customize the code for model loading and inference (see get_model_tokenizer_and_config and model_generate methods in inference_pipeline.py). Feel free to contact us if you have any other questions.