--max_examples seems to be broken

EleutherAI / elk

Keeping language models honest by directly eliciting knowledge encoded in their activations.

MIT License

175 stars 32 forks source link

--max_examples seems to be broken #257

Closed derpyplops closed 1 year ago

derpyplops commented 1 year ago

elk sweep --models huggyllama/llama-13b --datasets imdb --num_gpus 1 --net ccs is fine but

elk sweep --models huggyllama/llama-13b --datasets imdb --num_gpus 1 --net ccs --max_examples 10 10 has a bug where all the examples are the same as each other:

# elk/training/ccs_reporter.py#L275
x_neg, x_pos = self.norm(x_neg), self.norm(x_pos)
> torch.all(x_neg.eq(x_neg[0]))
True

derpyplops commented 1 year ago

cc @lauritowal

derpyplops commented 1 year ago

This was an issue specifically with https://github.com/EleutherAI/elk/pull/258, and isn't in main. Since it'll be fixed on that branch, closing this issue.