Closed ChristyKoh closed 1 year ago
The tests are structured as follows:
setup_elicit
creates and runs elk elicit with desired model/dataseteval_run
creates and runs eval, can specify transfer datasets and ccs/vinceval_assert_files_created
checks:
Does this sound reasonable? It's currently failing to assert directory creation for transfer evals.
The tests are structured as follows:
setup_elicit
creates and runs elk elicit with desired model/dataseteval_run
creates and runs eval, can specify transfer datasets and ccs/vinceval_assert_files_created
checks:
- vanilla eval results in modified eval.csv
- transfer eval creates new eval subdirectory structure and files
Does this sound reasonable? It's currently failing to assert directory creation for transfer evals.
taking a look at it!
Addresses #186