search

stanford-crfm / helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in Holistic Evaluation of Text-to-Image Models (HEIM) (https://arxiv.org/abs/2311.04287).
https://crfm.stanford.edu/helm
Apache License 2.0
1.8k stars 239 forks source link

[Issues #2439] Refactor the validation out of clients and add it to the request.py #2686

Closed SathvikNapa closed 1 week ago

SathvikNapa commented 2 months ago
yifanmai commented 2 months ago

Could you address the type checking issues?

In general, you should install the [dev] optional dependencies and run ./pre-commit.sh.

src/helm/clients/anthropic_client.py:290: error: Argument 1 to "get_dimensions" has incompatible type "Optional[str]"; expected "str"  [arg-type]
src/helm/clients/anthropic_client.py:339: error: Incompatible types (expression has type "Optional[str]", TypedDict item "text" has type "str")  [typeddict-item]
src/helm/clients/anthropic_client.py:342: error: Item "None" of "Optional[str]" has no attribute "strip"  [union-attr]
yifanmai commented 1 month ago

Please address:

Found 6 errors in 2 files (checked 586 source files)
src/helm/clients/anthropic_client.py:319: error: Argument "src" to "resize_image_to_max_file_size" has incompatible type "Optional[str]"; expected "str"  [arg-type]
src/helm/clients/anthropic_client.py:325: error: Argument 1 to "encode_base64" has incompatible type "Optional[str]"; expected "str"  [arg-type]
src/helm/clients/anthropic_client.py:339: error: Incompatible types (expression has type "Optional[str]", TypedDict item "text" has type "str")  [typeddict-item]