search

ChicagoHAI / active-example-selection

Active Example Selection for In-Context Learning (EMNLP'22)
MIT License
46 stars 2 forks source link

KeyError: 'label-coarse' #5

Closed Tizzzzy closed 3 months ago

Tizzzzy commented 3 months ago

Hi, Right now I am running this command python src/rl/main.py rl_configs/random-agent.yaml, and I changed dataset to trec. Here is how my random-agent.yaml looks like:

seed: 42
basedir: ./rl_outputs/
name: t=trec_s=42
group: search-offline
wandb_resume: true
agent: random
tests: []
agent_kwargs:
  train_steps: 2000
  save_every: 100
env: few-shot
env_kwargs:
  named_features: true
  model: gpt2-medium
  model_kwargs:
    batch_size: 8
    calibrate: true
  max_steps: 4
  train_subset_samples: 100
  eval_subset_samples: 80
  action_repr:
  - logits
  - probs
  - logits_stats
  - probs_stats
  - entropy
  - perplexity
  state_repr:
  - curr_step
  - max_steps
  - last_label
  - val_dist
  - val_dist_entropy
  - val_dist_stats
  - perplexity
  task: trec
  max_feasible_steps: 8
output_dir: outputs/trec-random

After I run the command I got this error:

(active-example-selection) [conda] [lh599@corfu:active-example-selection]$ CUDA_VISIBLE_DEVICES=1 python src/rl/main.py rl_configs/random-agent.yaml
24-07-24 20:46 - root - INFO - logging to outputs/trec-random/debug.txt.
wandb: Currently logged in as: ligongh (cqueen). Use `wandb login --relogin` to force relogin
wandb: wandb version 0.17.5 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.13.5
wandb: Run data is saved locally in /research/cbim/medical/lh599/research/ruijiang/Dong/demonstration_selection/active-example-selection/wandb/run-20240724_204650-search-offline.t=trec_s=42
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run t=trec_s=42
wandb: ⭐️ View project at https://wandb.ai/cqueen/active-example-selection
wandb: 🚀 View run at https://wandb.ai/cqueen/active-example-selection/runs/search-offline.t%3Dtrec_s%3D42
24-07-24 20:46 - __main__ - INFO - {'seed': 42, 'basedir': './rl_outputs/', 'name': 't=trec_s=42', 'group': 'search-offline', 'wandb_resume': True, 'agent': 'random', 'tests': [], 'full_tests': [], 'agent_kwargs': {'train_steps': 2000, 'save_every': 100}, 'env': 'few-shot', 'env_kwargs': {'named_features': True, 'model': 'gpt2-medium', 'model_kwargs': {'batch_size': 8, 'calibrate': True}, 'max_steps': 4, 'train_subset_samples': 100, 'eval_subset_samples': 80, 'action_repr': ['logits', 'probs', 'logits_stats', 'probs_stats', 'entropy', 'perplexity'], 'state_repr': ['curr_step', 'max_steps', 'last_label', 'val_dist', 'val_dist_entropy', 'val_dist_stats', 'perplexity'], 'task': 'trec', 'max_feasible_steps': 8}, 'output_dir': 'outputs/trec-random'}
24-07-24 20:46 - __main__ - INFO - Seed: 42
24-07-24 20:46 - root - INFO - re-seeding with seed 42.
24-07-24 20:46 - prompting.data_utils - INFO - generating datasets using seed 42
24-07-24 20:46 - prompting.data_utils - INFO - dataset hash=5943735295261645614663961736908204131290335488058648383887579511042200465745866929066331613141860222860699017704096835378366022810171539918482872107096762
24-07-24 20:46 - prompting.models - INFO - Setting batch_size=8
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
24-07-24 20:46 - prompting.models - INFO - Initializing gpt2-medium
24-07-24 20:46 - prompting.models - WARNING - Cannot find matching id for Abbreviation, using prefix ĠAb
24-07-24 20:46 - prompting.models - INFO - Labels: ['Description', 'Entity', 'Abbreviation', 'Person', 'Number', 'Location']
24-07-24 20:46 - rl.environment - INFO - train_dataset size=100
24-07-24 20:46 - rl.environment - INFO - val_dataset size=100
24-07-24 20:46 - rl.environment - INFO - test_dataset size=500
24-07-24 20:46 - rl.environment - INFO - sampling 100 out of 100 for each training episode
24-07-24 20:46 - rl.environment - INFO - sampling 80 out of 100 for each validation episode
24-07-24 20:46 - rl.agents.random_agent - INFO - no existing checkpoints, train from scratch...
  0%|                                                                                                        | 0/2000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/research/cbim/medical/lh599/research/ruijiang/Dong/demonstration_selection/active-example-selection/src/rl/main.py", line 73, in <module>
    main()
  File "/research/cbim/medical/lh599/research/ruijiang/Dong/demonstration_selection/active-example-selection/src/rl/main.py", line 54, in main
    agent.train()
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/agents/random_agent.py", line 124, in train
    self.rollout()
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/agents/random_agent.py", line 73, in rollout
    state = env.reset()
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 314, in reset
    self.prev_acc = self.zero_shot_acc
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 216, in zero_shot_acc
    return self.zero_shot_acc_train
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/functools.py", line 993, in __get__
    val = self.func(instance)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 284, in zero_shot_acc_train
    acc = self.evaluate_accuracy([], "train")
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 373, in evaluate_accuracy
    eval_prompts, eval_cali_prompts = self.proc.create_prompts(
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 161, in create_prompts
    prompt = Prompt(training_prompt + self.fill_eval_template(eval_example))
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 99, in fill_eval_template
    fields = self.convert_example_to_template_fields(example)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 306, in convert_example_to_template_fields
    label_text = self.labels[example["label-coarse"]]
KeyError: 'label-coarse'
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: Synced t=trec_s=42: https://wandb.ai/cqueen/active-example-selection/runs/search-offline.t%3Dtrec_s%3D42
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240724_204650-search-offline.t=trec_s=42/logs
y0mingzhang commented 3 months ago

The keys in TREC were updated to ['text', 'coarse_label', 'fine_label'] at some point after the code was written. You could probably fix this by renaming things.

y0mingzhang commented 3 months ago

Also a note that I am archiving this repository since the code was written two years ago, and is meant to be used as is.