search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.76k stars 234 forks source link

Fix backports.zoneinfo and pkg_resources breakages in main #2770

Closed yifanmai closed 1 week ago

yifanmai commented 1 week ago

Fix error on Python 3.10 by removing backports.zoneinfo from requirements.txt:

Failed to build backports.zoneinfo
ERROR: Could not build wheels for backports.zoneinfo, which is required to install pyproject.toml-based projects

Also fix this error by switching test.yml to not use virtualenv:

________________ TestQ16.test_is_appropriate_single_appropriate ________________

self = <helm.benchmark.metrics.image_generation.q16.test_q16.TestQ16 object at 0x7f68f0bd6320>

    def test_is_appropriate_single_appropriate(self):
        image_path: str = os.path.join(self._base_path, "sample_appropriate.jpg")
>       assert not self._q16_detector.is_inappropriate(image_path)

src/helm/benchmark/metrics/image_generation/q16/test_q16.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/helm/benchmark/metrics/image_generation/q16/q16_toxicity_detector.py:56: in is_inappropriate
    self._load_models()
src/helm/benchmark/metrics/image_generation/q16/q16_toxicity_detector.py:36: in _load_models
    self._clip_wrapper = ClipWrapper(self._device)
src/helm/benchmark/metrics/image_generation/q16/q16_toxicity_detector.py:67: in __init__
    import clip
venv/lib/python3.10/site-packages/clip/__init__.py:1: in <module>
    from .clip import *
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    import hashlib
    import os
    import urllib
    import warnings
    from typing import Any, Union, List
>   from pkg_resources import packaging
E   ImportError: cannot import name 'packaging' from 'pkg_resources' (/home/runner/work/helm/helm/venv/lib/python3.10/site-packages/pkg_resources/__init__.py)

venv/lib/python3.10/site-packages/clip/clip.py:6: ImportError