search

alan-turing-institute / prompto

An open source library for asynchronous querying of LLM endpoints
https://alan-turing-institute.github.io/prompto/
MIT License
20 stars 1 forks source link

Evaluation functions #94

Closed EdwinB12 closed 2 months ago

EdwinB12 commented 2 months ago

Very simple application of using an evaluation function in prompto.

review-notebook-app[bot] commented 2 months ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

EdwinB12 commented 2 months ago

This would work better as a method in experiment. The user would run it outside of experiment.process()

EdwinB12 commented 2 months ago

Restriction on passed function is it must take in a prompt dictionary and it must return a prompt dictionary

EdwinB12 commented 2 months ago

Should support a list/tuple of functions. Don't support arguments. Encourage the user to use the prompt dictionary to parameterise.

EdwinB12 commented 2 months ago

This has ended up being a very bare bones application of this and i'm not sure what value it actually adds over just running an evaluation function on the completed responses dictionary saved to disk after called .process().

codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 52.41%. Comparing base (cf15ce4) to head (824734a). Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
src/prompto/experiment.py 80.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #94 +/- ## =========================================== + Coverage 35.67% 52.41% +16.74% =========================================== Files 38 38 Lines 1962 1984 +22 =========================================== + Hits 700 1040 +340 + Misses 1262 944 -318 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.