search

mozilla-ai / lm-buddy

Your buddy in the (L)LM space.
Apache License 2.0
63 stars 3 forks source link

Add input data + predictions to output file #103

Closed aittalam closed 4 months ago

aittalam commented 5 months ago

What's changing

Added the possibility of storing both input data and predictions to the output json file

veekaybee commented 5 months ago

How big do these end up being and what does the final output file look like in this case? it would be nice to have some output examples!

aittalam commented 4 months ago

How big do these end up being and what does the final output file look like in this case? it would be nice to have some output examples!

Great point! Here's an output example. If one adds everything the file might become quite large: this one is 12KB for 10 samples, the dialogsum validation which has 500 samples should become about 600KB. This is the reason why I wanted to add the option to only have the eval metrics saved. In any case, if the input size is D, the output dataset with everything will be of size E = D + e where e is typically quite smaller than D as the new data are metrics and summarized texts.

eval_results.json

aittalam commented 4 months ago

The code looks good but I was curious about the outputs + also looks like tests need a bump

For the integration tests: I saw the error 

"cannot import name 'packaging' from 'pkg_resources'"

Occurs a few times and every time it starts from some ray code. So I thought perhaps the problem was not in setuptools as I originally thought (see the plethora of posts here) but rather in ray. I looked into one of the failing scripts (dashboard/modules/dashboard_sdk.py) and found that the latest update to it was exactly removing the deprecated pkg_resources in favor of importlib.metadata.

This happened in ray version 2.24.0 and currently we have 2.9.3 set in pyproject.toml. @chainlink is there anything against us spinning up ray clusters with a more recent ray version, so we can update it here too?