search

facebookresearch / clutrr

Diagnostic benchmark suite to explicitly test logical relational reasoning on natural language
Other
90 stars 14 forks source link

Unable to generate clutrr data #23

Open aadityanaik opened 1 month ago

aadityanaik commented 1 month ago

I am trying to generate clutrr data using the command given in the README:

python main.py --train_tasks 1.2 --test_tasks 1.2 --train_rows 500 --test_rows 10 --equal --holdout --use_mturk_template --data_name "Robust Reasoning - clean - AMT" --unique_test_pattern

However, I encounter this error:

2024-09-20 03:41:00,047 - INFO - Data generation started for configurations :
2024-09-20 03:41:00,047 - INFO -
togrep : ['--train_tasks', '1.2', '--test_tasks', '1.2', '--train_rows', '500', '--test_rows', '10', '--equal', '--holdout', '--use_mturk_template', '--data_name', 'Robust Reasoning - clean - AMT', '--unique_test_pattern']

Downloading placeholder data
sha256 : 58ad947162d2293c2259264e3913235e1c3b73e807a8db0fcd867eb09c972187
checking ...
Traceback (most recent call last):
  File "/home/aadityanaik/clutrr/clutrr/main.py", line 383, in <module>
    cl = Clutrr(args)
         ^^^^^^^^^^^^
  File "/home/aadityanaik/clutrr/clutrr/main.py", line 62, in __init__
    self.setup()
  File "/home/aadityanaik/clutrr/clutrr/main.py", line 370, in setup
    raise AssertionError("downloaded corrupt data, sha256 doesn't match")
AssertionError: downloaded corrupt data, sha256 doesn't match

This seems to stem from trying to download the cleaned_placeholders.zip file, and the download link for that doesn't seem to be working. Is there any other place I can download this file from?

xywen97 commented 1 day ago

same problem, have you solved this problem?