search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.86k stars 243 forks source link

HEIMHumanEvalScenario requires permissions to download data from codalab #2865

Open slymane opened 1 month ago

slymane commented 1 month ago

Hi folks, thanks for the work here! I'm trying to use the HEIMHumanEvalScenario class but run into an error after get_instances() is called. Specifically, https://worksheets.codalab.org/rest/bundles/0x502d646c366c4f1d8c4a2ccf163b958f/contents/blob/ returns ERROR 403: Forbidden when using wget to download data during the HEIMHumanEvalScenario.get_instances() call: https://github.com/stanford-crfm/helm/blob/60a58658b4648d7b027869a6b22fec1af2eaa855/src/helm/benchmark/scenarios/vision_language/heim_human_eval_scenario.py#L62

Visiting the URL manually shows a forbidden error specifying that read permissions are needed. Is this intended behavior?

yifanmai commented 1 month ago

@teetone could you answer regarding HEIM?