Question about running stage 3 script

[2003pro]

Following the script provided in the second step of "Selecting data for a task" in your readme, I have a command line that needs to be run as shown below:

./less/scripts/data_selection/matching.sh ../grads/llama2-7b-p0.05-lora-seed3/{}-ckpt{}-adam/dim8192 "flan_v2 cot dolly oasst1" "422 845 1268 1688" "1.681734880879658e-05 1.4844950172237425e-05 1.2464498996972609e-05 9.999999999999999e-06" ../grads/llama2-7b-p0.05-lora-seed3/{}-ckpt{}-sgd/dim8192 "tydiqa" "../selected_data"

However, after running it, I encountered the following error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/export/project/jzhanggr/qyx/LESS/less/data_selection/matching.py", line 58, in <module>
    validation_info = torch.load(validation_path)
  File "/export/project/jzhanggr/miniconda3/lib/python3.11/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/export/project/jzhanggr/miniconda3/lib/python3.11/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/export/project/jzhanggr/miniconda3/lib/python3.11/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
IsADirectoryError: [Errno 21] Is a directory: '../grads/llama2-7b-p0.05-lora-seed3/tydiqa-ckpt422-sgd/dim8192'

This location appears to be a file, but upon inspection, there are three files in that folder named all_orig.pt, all_unormalized.pt, and grads-9.pt. I wonder if I should be loading grads-9.pt at this point. Your assistance would be greatly appreciated.

[2003pro]

BTW, I met a similar issue on Adam version:

IsADirectoryError: [Errno 21] Is a directory: '../grads/llama2-7b-p0.05-lora-seed3/flan_v2-ckpt422-adam'

[xiamengzhou]

You should be using all_orig.pt, which is the normalized gradients! Sorry for the confusion.

[shangqing-liu]

@2003pro Hi have you successfully run the code?