search

EleutherAI / elk

Keeping language models honest by directly eliciting knowledge encoded in their activations.
MIT License
178 stars 33 forks source link

Multiple choice datasets encounter a tensor error #241

Open ethanedwards opened 1 year ago

ethanedwards commented 1 year ago

Multiple choice datasets such as "glue:mnli" returns a tensor error.

The command

elk elicit gpt2 "glue:mnli" --num_gpus 2

returns this output

Using 2 of 8 GPUs: [0, 4] glue:mnli using 'train' for training and 'validation_matched' for validation Found cached dataset generator (/home/ethan/.cache/huggingface/datasets/generator/default-a729374a01df289c/0.0.0) Found cached dataset generator (/home/ethan/.cache/huggingface/datasets/generator/default-5453548fe42aba36/0.0.0) Output directory at /home/ethan/elk-reporters/gpt2/glue:mnli/kind-golick 0%| | 0/13 [00:08<?, ?it/s] multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/training/train.py", line 133, in apply_to_layer evaluate_preds(val_gt, val_credences, mode).to_dict(), File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/metrics/eval.py", line 75, in evaluate_preds auroc = roc_auc_ci(to_one_hot(y_true, c).long(), y_logits) File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/metrics/roc_auc.py", line 103, in roc_auc_ci raise ValueError("y_true and y_pred should be 1D or 2D tensors") ValueError: y_true and y_pred should be 1D or 2D tensors """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/mnt/ssd-1/spar/ethan/miniconda3/bin/elk", line 8, in sys.exit(run()) File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/main.py", line 27, in run run.execute() File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/main.py", line 19, in execute return self.command.execute() File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/run.py", line 98, in execute self.apply_to_layers(func=func, num_devices=num_devices) File "/mnt/ssd-1/spar/ethan/elkethan2/elk/elk/run.py", line 175, in apply_to_layers for df_dict in tqdm(mapper(func, layers), total=len(layers)): File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/site-packages/tqdm/std.py", line 1195, in iter for obj in iterable: File "/mnt/ssd-1/spar/ethan/miniconda3/lib/python3.10/multiprocessing/pool.py", line 873, in next raise value ValueError: y_true and y_pred should be 1D or 2D tensors