issues
search
EleutherAI
/
elk
Keeping language models honest by directly eliciting knowledge encoded in their activations.
MIT License
178
stars
33
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Allow for arbitrary hyperparameter selection for sweep
#198
norabelrose
closed
1 year ago
2
Cluster bootstrap for metrics; refactor metric computations into evaluate_preds
#197
norabelrose
closed
1 year ago
0
Fix acc for `supervised` (in the same way as #195)
#196
AlexTMallen
closed
1 year ago
1
Fix accuracy computation in `Reporter`
#195
AlexTMallen
closed
1 year ago
0
Fix swapped labels and probs in calibration error metric update
#194
AlexTMallen
closed
1 year ago
0
Add way of specifying nondefault hparams that are shared across sweep
#193
lauritowal
closed
1 year ago
1
Add transfer eval to sweep
#192
lauritowal
closed
1 year ago
1
Sweep MVP
#191
norabelrose
closed
1 year ago
2
Blazing fast bootstrap stderrs for AUROC
#190
norabelrose
closed
1 year ago
0
Multiple datasets refactor
#189
norabelrose
closed
1 year ago
1
fix multi eval behavior
#188
ChristyKoh
closed
1 year ago
1
add custom_ds_n CLI arg
#187
reaganjlee
closed
1 year ago
2
Smoke tests for elk eval command
#186
norabelrose
closed
1 year ago
4
simplify eval command syntax
#185
ChristyKoh
closed
1 year ago
1
spar mt prompt invar
#184
ChristyKoh
closed
1 year ago
0
fix label assignment causing multiclass bug
#183
ChristyKoh
closed
1 year ago
0
Prevent invalidation of the hidden state cache when num_gpus changes
#182
norabelrose
closed
1 year ago
0
Add portuguese templates for boolq, ag_news, imdb
#181
ChristyKoh
closed
1 year ago
0
Add ELLMo Integration
#180
Kyle1668
closed
1 year ago
2
Support multiple choice datasets
#179
norabelrose
closed
1 year ago
0
[pre-commit.ci] pre-commit autoupdate
#178
pre-commit-ci[bot]
closed
1 year ago
0
Force spawn start method
#177
norabelrose
closed
1 year ago
0
[Draft/WIP] Support VINC with ELMo
#176
Kyle1668
closed
1 year ago
0
Use model name and dataset to organize reporters in `elicit`
#175
norabelrose
closed
1 year ago
0
save eval runs to separate subfolders by target dataset
#174
ChristyKoh
closed
1 year ago
0
save elk eval runs separately
#173
ChristyKoh
closed
1 year ago
0
Combine prompts to evaluate multilingual prompt invariance
#172
ChristyKoh
closed
1 year ago
0
Evaluate baseline when running eval
#171
lauritowal
closed
1 year ago
1
Make normalization a property of `Reporter`; support eval with only one split
#170
AlexTMallen
closed
1 year ago
0
Support exponential moving averages for the covariance statistics on EigenReporter
#169
norabelrose
opened
1 year ago
0
Turn on import sorting
#168
norabelrose
closed
1 year ago
0
Added confidence, invariance, consistency metrics
#167
ss-waree
closed
1 year ago
0
check is streamable
#166
AlexTMallen
closed
1 year ago
6
Store final layer LM output and record AUROC and acc
#165
norabelrose
closed
1 year ago
0
Refactor & rename lanczos_eigsh for convergence, correctness, & speed
#164
norabelrose
closed
1 year ago
0
Hyperparameter sweeps with Optuna or Ray Tune
#163
norabelrose
closed
1 year ago
1
[pre-commit.ci] pre-commit autoupdate
#162
pre-commit-ci[bot]
closed
1 year ago
0
Add LR stats to transfer eval
#161
lauritowal
closed
1 year ago
0
split max_examples between processes
#160
AlexTMallen
closed
1 year ago
0
Allow min_memory to be passed by CLI
#159
thejaminator
closed
1 year ago
1
Revert "Multi datasets"
#158
lauritowal
closed
1 year ago
3
Gather eval stats
#157
reaganjlee
closed
1 year ago
2
update README.md
#156
lauritowal
closed
1 year ago
0
Create dataclasses for writing to CSV, refactor CSV logging, fix eval CSV columns
#155
thejaminator
closed
1 year ago
0
Replace flake with ruff
#154
thejaminator
closed
1 year ago
0
train_reporter should return a dataclass of stats instead of a list
#153
thejaminator
closed
1 year ago
0
Make num_heads property on EigenReporter accessible via CLI
#152
norabelrose
opened
1 year ago
0
Hello World
#151
norabelrose
closed
1 year ago
0
Create devcontainer
#150
derpyplops
closed
1 year ago
1
Smoke tests with tiny gpt2, fix CCSReporter
#149
thejaminator
closed
1 year ago
0
Previous
Next