search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
825 stars 219 forks source link

Update README.md #204

Open AnitaLiu98 opened 8 months ago

AnitaLiu98 commented 8 months ago

Fix the issue where solutions generated after executing the bash script are not being saved.

loubnabnl commented 8 months ago

The generations are saved by default in the generation_only mode https://github.com/bigcode-project/bigcode-evaluation-harness/blob/e54f33d093f342ffc0c5c057910c00aa2081515d/main.py#L389

Vipitis commented 8 months ago

I am also experiencing this. If you set --generation_only and --save_generations_path "./gens.json" but not --save_generations It does not write to file. Several examples and the README should be adjusted, else work might be lost.

problematic behaviour is this line: https://github.com/bigcode-project/bigcode-evaluation-harness/blob/e54f33d093f342ffc0c5c057910c00aa2081515d/bigcode_eval/evaluator.py#L117

seems like an easy fix would be setting 

self.args.save_generations = True if self.args.generations_only else self.args.save_generations

Or 

if self.args.save_generations or self.args.generation_only:
loubnabnl commented 8 months ago

Ah I missed that, feel free to open a PR that adds if self.args.save_generations or self.args.generation_only: @Vipitis . Thanks!

Vipitis commented 8 months ago

Ah I missed that, feel free to open a PR that adds if self.args.save_generations or self.args.generation_only: @Vipitis . Thanks!

Will do, but won't be able to submit it until next week.