search

ChicagoHAI / active-example-selection

Active Example Selection for In-Context Learning (EMNLP'22)
MIT License
45 stars 2 forks source link

ValueError: Invalid pattern: '**' can only be an entire path component #4

Closed Tizzzzy closed 2 months ago

Tizzzzy commented 2 months ago

Hi, I followed every step in the README and did not change any code. I did python src/rl/main.py rl_configs/random-agent.yaml, and python src/rl/main.py rl_configs/agnews-same-task.yaml. Now I am trying to run python src/rl/main.py rl_configs/agnews-transfer.yaml. However, I get this error:

(active-example-selection) [conda] [lh599@corfu:active-example-selection]$ CUDA_VISIBLE_DEVICES=1 python src/rl/main.py rl_configs/agnews-transfer.yaml
24-07-23 14:37 - root - INFO - logging to outputs/agnews-transfer/debug.txt.
wandb: Currently logged in as: ligongh (cqueen). Use `wandb login --relogin` to force relogin
wandb: wandb version 0.17.5 is available!  To upgrade, please run:
wandb:  $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.13.5
wandb: Run data is saved locally in /research/cbim/medical/lh599/research/ruijiang/Dong/demonstration_selection/active-example-selection/wandb/run-20240723_143737-k8dorcfa
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run default
wandb: ⭐️ View project at https://wandb.ai/cqueen/active-example-selection
wandb: 🚀 View run at https://wandb.ai/cqueen/active-example-selection/runs/k8dorcfa
24-07-23 14:37 - __main__ - INFO - {'seed': 42, 'basedir': '.outputs', 'name': 'default', 'group': 'ungrouped', 'wandb_resume': False, 'agent': 'dqn', 'tests': [], 'full_tests': [{'env_kwargs': {'dataset_mode': 'unlabeled'}, 'test_kwargs': {'eval_prefix': 'unlabeled-best', 'load': 'best'}}], 'agent_kwargs': {'train_steps': 8000, 'save_every': 800, 'eval_every': 800, 'batch_size': 16, 'max_grad_norm': 1.0, 'lr': 0.0003, 'weight_decay': 0.001, 'network_params': {'task_count': 4, 'dropout': 0.25}, 'val_rounds': 12, 'target_update_every': 400, 'optimization_steps_per_train_step': 16, 'replay_memory_size': 50000, 'offline_steps': 12000, 'cql_loss_weight': 0.2, 'load_transitions': ['outputs/amazon-random/ckpts/transitions_2000.ckpt', 'outputs/sst-2-random/ckpts/transitions_2000.ckpt', 'outputs/trec-random/ckpts/transitions_2000.ckpt']}, 'env': 'multi-dataset', 'env_kwargs': {'model': 'gpt2-medium', 'model_kwargs': {'batch_size': 8, 'calibrate': True}, 'max_steps': 4, 'train_subset_samples': 100, 'eval_subset_samples': 100, 'max_feasible_steps': 8, 'action_repr': ['entropy', 'probs_stats', 'perplexity'], 'state_repr': ['curr_step', 'perplexity'], 'train_env_configs': [{'task': 'amazon'}, {'task': 'sst-2'}, {'task': 'trec'}], 'test_env_config': {'task': 'agnews'}}, 'output_dir': 'outputs/agnews-transfer'}
24-07-23 14:37 - __main__ - INFO - Seed: 42
24-07-23 14:37 - root - INFO - re-seeding with seed 42.
24-07-23 14:37 - prompting.data_utils - INFO - generating datasets using seed 42
Downloading builder script: 4.11kB [00:00, 7.09MB/s]
Downloading metadata: 1.68kB [00:00, 5.74MB/s]
Downloading and preparing dataset amazon_polarity/amazon_polarity (download: 656.45 MiB, generated: 1.66 GiB, post-processed: Unknown size, total: 2.30 GiB) to /research/cbim/vast/lh599/.cache/huggingface/datasets/amazon_polarity/amazon_polarity/3.0.0/a27b32b7e7b88eb274a8fa8ba0f654f1fe998a87c22547557317793b5d2772dc...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████| 688M/688M [00:53<00:00, 12.7MB/s]
Dataset amazon_polarity downloaded and prepared to /research/cbim/vast/lh599/.cache/huggingface/datasets/amazon_polarity/amazon_polarity/3.0.0/a27b32b7e7b88eb274a8fa8ba0f654f1fe998a87c22547557317793b5d2772dc. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 140.71it/s]
24-07-23 14:41 - prompting.data_utils - INFO - subsampling test set to 10000 examples
24-07-23 14:41 - prompting.data_utils - INFO - dataset hash=10680324770379770683240110755448239101172121195275020032919222970444757180295230544994057011964013618886415389314056882105957921289081249251806812879168192
24-07-23 14:41 - prompting.models - INFO - Setting batch_size=8
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
24-07-23 14:41 - prompting.models - INFO - Initializing gpt2-medium
24-07-23 14:41 - prompting.models - INFO - Labels: ['negative', 'positive']
24-07-23 14:41 - rl.environment - INFO - train_dataset size=100
24-07-23 14:41 - rl.environment - INFO - val_dataset size=100
24-07-23 14:41 - rl.environment - INFO - test_dataset size=10000
24-07-23 14:41 - rl.environment - INFO - sampling 100 out of 100 for each training episode
24-07-23 14:41 - rl.environment - INFO - sampling 100 out of 100 for each validation episode
24-07-23 14:41 - prompting.data_utils - INFO - generating datasets using seed 42
Traceback (most recent call last):
  File "/research/cbim/medical/lh599/research/ruijiang/Dong/demonstration_selection/active-example-selection/src/rl/main.py", line 73, in <module>
    main()
  File "/research/cbim/medical/lh599/research/ruijiang/Dong/demonstration_selection/active-example-selection/src/rl/main.py", line 50, in main
    env = env_type(seed=seed, **conf.env_kwargs)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 875, in __init__
    self.train_envs = [
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 876, in <listcomp>
    FewShotEnvironment(task_id=task_id, **task_kwargs)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/rl/environment.py", line 100, in __init__
    processor = PROCESSORS[task](seed, dataset_mode)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 253, in __init__
    self.generate_datasets(seed, mode)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 35, in generate_datasets
    if len(self.train_split) < 1200:
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/functools.py", line 993, in __get__
    val = self.func(instance)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 20, in train_split
    return self.dataset["train"]
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/functools.py", line 993, in __get__
    val = self.func(instance)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/prompting/data_utils.py", line 16, in dataset
    return load_dataset(self.dataset_name)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/load.py", line 1664, in load_dataset
    builder_instance = load_dataset_builder(
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/load.py", line 1490, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/load.py", line 1242, in dataset_module_factory
    raise e1 from None
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/load.py", line 1223, in dataset_module_factory
    return HubDatasetModuleFactoryWithoutScript(
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/load.py", line 846, in get_module
    else get_patterns_in_dataset_repository(hfh_dataset_info)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/data_files.py", line 471, in get_patterns_in_dataset_repository
    return _get_data_files_patterns(resolver)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/data_files.py", line 99, in _get_data_files_patterns
    data_files = pattern_resolver(pattern)
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/datasets/data_files.py", line 303, in _resolve_single_pattern_in_dataset_repository
    glob_iter = [PurePath(filepath) for filepath in fs.glob(PurePath(pattern).as_posix()) if fs.isfile(filepath)]
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/fsspec/spec.py", line 606, in glob
    pattern = glob_translate(path + ("/" if ends_with_sep else ""))
  File "/research/cbim/medical/lh599/research/ruijiang/miniconda/envs/active-example-selection/lib/python3.9/site-packages/fsspec/utils.py", line 734, in glob_translate
    raise ValueError(
ValueError: Invalid pattern: '**' can only be an entire path component
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb: Synced default: https://wandb.ai/cqueen/active-example-selection/runs/k8dorcfa
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240723_143737-k8dorcfa/logs
y0mingzhang commented 2 months ago

What's your datasets and fsspec versions?

Tizzzzy commented 2 months ago 
Name: datasets
Version: 2.1.0
Name: fsspec
Version: 2024.2.0

FYI, it seems like the provided code in rl_configs/random-agent.yaml, and rl_configs/agnews-same-task.yaml are all for agnews dataset. Does it means in order to run python src/rl/main.py rl_configs/agnews-transfer.yaml, I have to change dataset in random-agent.yaml and agnews-same-task.yaml to Amazon, SST-2 and TREC datasets, and run command for each dataset? If so, how can I run Amazon, SST-2 and TREC datasets?

y0mingzhang commented 2 months ago

I cannot reproduce, but the issue seems related to https://github.com/huggingface/datasets/issues/6737#issuecomment-2107336816, and you can probably fix the issue by upgrading datasets.

To use other datasets, update the task field in configs. https://github.com/ChicagoHAI/active-example-selection/blob/f51558ef675cdb59c317728ca26431fd0065f376/rl_configs/random-agent.yaml#L36

Here are the aliases of supported datasets: https://github.com/ChicagoHAI/active-example-selection/blob/f51558ef675cdb59c317728ca26431fd0065f376/src/prompting/__init__.py#L25-L30