Using --extra_eval_params will override args of the same name specified in the eval .yaml, but the updated values will not be logged in the spec, the original values will be logged instead. This PR fixes this problem; we just update eval_spec.args with the new values.
e.g. running oaieval dummy make-me-pay --extra_eval_params turn_cap=1 previously lead to "turn_cap": 5 being logged in the spec, since this is the default value.
In this branch, running the same command leads to "turn_cap": 1 being logged in the spec.
Using
--extra_eval_paramswill override args of the same name specified in the eval .yaml, but the updated values will not be logged in the spec, the original values will be logged instead. This PR fixes this problem; we just updateeval_spec.argswith the new values.e.g. running
oaieval dummy make-me-pay --extra_eval_params turn_cap=1previously lead to"turn_cap": 5being logged in the spec, since this is the default value. In this branch, running the same command leads to"turn_cap": 1being logged in the spec.