Rule_qa data are missing

[mistycheney]

tasks/rule_qa/train.tsv is empty.

[neelguha]

That task was intended to be a zero-shot task with no training samples!

[cbrew]

sure, but if that is the case, and there are no training examples the comment in the documentation that the training data contains examples of possible answers needs to be revised. The paper does describe how the evaluation was done, which is informative. But it also says, in appendix E, that the answer key is available from the website. I didn't find that. Is it somewhere non-obvious?

[neelguha]

• Apologies for the confusion on the documentation. We've updated it to clarify that rule_qa and scalr are the two tasks where there's no training samples provided.
• For rule_qa, answers can be found on the huggingface repository: https://huggingface.co/datasets/nguha/legalbench/blob/main/data/rule_qa/test.tsv

If you're looking for answers for the chain-of-thought style generations we evaluated for the rule-application tasks, those can be found here: https://hazyresearch.stanford.edu/legalbench/getting-started/.

Please let me know if I'm misunderstanding something!