search

neuralmagic / guidellm

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
Apache License 2.0
158 stars 11 forks source link

Add summary metrics to saved json file #46

Closed anmarques closed 1 month ago

anmarques commented 2 months ago

At the end of the run GuideLLM will print summary metrics that are computed from the raw results, but these are not currently saved anywhere.

This PR adds these metrics as serializable properties of the TextGenerationBenchmark class. Most of the metrics are already declared as properties, but are not serializable. Adding the @computed_field decorator is enough in these cases. Other properties were added to complete the list.

EricAntidot commented 2 months ago

This is a nice addition we are expecting this to be merged.

EricAntidot commented 2 months ago

@anmarques In order to calm down the mypy check you should add # type: ignore[misc] to the @computed_field annotation line. cf: https://github.com/python/mypy/issues/1362 or pydantic doc as well: https://docs.pydantic.dev/2.0/usage/computed_fields/

anmarques commented 2 months ago

@anmarques In order to calm down the mypy check you should add # type: ignore[misc] to the @computed_field annotation line. cf: python/mypy#1362 or pydantic doc as well: https://docs.pydantic.dev/2.0/usage/computed_fields/

Thanks for the tip!

anmarques commented 1 month ago

I don't have instructions to convert to csv readily available. Will look into it in a following PR. Will land this one to unblock anyone needing the information saved.