EleutherAI / elk

Keeping language models honest by directly eliciting knowledge encoded in their activations.
MIT License
178 stars 33 forks source link

Don't try loading models from the cwd ever #223

Closed norabelrose closed 1 year ago

norabelrose commented 1 year ago

This fixes a very annoying quirk of HuggingFace from_pretrained methods: if you try to load, say, huggyllama/llama-7b when there's a folder named huggyllama/llama-7b in the current working directory, it'll try to load from that folder instead of the HF Hub (or the relevant cache dir). We basically never want this behavior, and it becomes a problem when you happen to be cded into the elk-reporters directory for example.

This PR fixes this with a little hack: we just move the offending local directory to a temp dir for a short time while we call Auto<whatever>.from_pretrained, then move it back. AFAICT there's not a better way to do this.

Also, as a bonus, this PR makes the change that Christy and I talked about like a week ago, where datasets like "super_glue boolq" are now named like super_glue:boolq so that you don't have to wrap it in double quotes on the command line

AlexTMallen commented 1 year ago

I did mkdir gpt2 from the elk repo root, and then got the following result:


❯ elk elicit gpt2 amazon_polarity --max_examples 10 --debug
Traceback (most recent call last):
  File "/mnt/ssd-2/spar/alexm/miniconda3/envs/elk/bin/elk", line 8, in <module>
    sys.exit(run())
  File "/mnt/ssd-2/spar/alexm/elk/elk/__main__.py", line 27, in run
    run.execute()
  File "/mnt/ssd-2/spar/alexm/elk/elk/__main__.py", line 19, in execute
    return self.command.execute()
  File "/mnt/ssd-2/spar/alexm/elk/elk/run.py", line 50, in execute
    self.datasets = [
  File "/mnt/ssd-2/spar/alexm/elk/elk/run.py", line 51, in <listcomp>
    extract(
  File "/mnt/ssd-2/spar/alexm/elk/elk/extraction/extraction.py", line 310, in extract
    with temporary_dir_move(cfg.model):
  File "/mnt/ssd-2/spar/alexm/miniconda3/envs/elk/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/mnt/ssd-2/spar/alexm/elk/elk/utils/data_utils.py", line 64, in temporary_dir_move
    path.rename(dest)
  File "/mnt/ssd-2/spar/alexm/miniconda3/envs/elk/lib/python3.10/pathlib.py", line 1234, in rename
    self._accessor.rename(self, target)
OSError: [Errno 18] Invalid cross-device link: 'gpt2' -> '/tmp/tmpwn7ysohc/gpt2'```