EleutherAI / elk

Keeping language models honest by directly eliciting knowledge encoded in their activations.
MIT License
178 stars 33 forks source link

Eval tests #204

Closed ChristyKoh closed 1 year ago

ChristyKoh commented 1 year ago

Addresses #186

ChristyKoh commented 1 year ago

The tests are structured as follows:

  1. setup_elicit creates and runs elk elicit with desired model/dataset
  2. eval_run creates and runs eval, can specify transfer datasets and ccs/vinc
  3. eval_assert_files_created checks:
    • vanilla eval results in modified eval.csv
    • transfer eval creates new eval subdirectory structure and files

Does this sound reasonable? It's currently failing to assert directory creation for transfer evals.

CLAassistant commented 1 year ago

CLA assistant check
All committers have signed the CLA.

thejaminator commented 1 year ago

The tests are structured as follows:

  1. setup_elicit creates and runs elk elicit with desired model/dataset
  2. eval_run creates and runs eval, can specify transfer datasets and ccs/vinc
  3. eval_assert_files_created checks:
  • vanilla eval results in modified eval.csv
  • transfer eval creates new eval subdirectory structure and files

Does this sound reasonable? It's currently failing to assert directory creation for transfer evals.

taking a look at it!