search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
152 stars 40 forks source link

feat: add SaveStrategy to allow flexibility in saving localized evaluation outputs #281

Closed keerthanvasist closed 1 month ago

keerthanvasist commented 1 month ago

Add SaveStrategy to allow flexibility in saving localized evaluation outputs

Description of changes: Introduces a new class SaveStrategy that allows users define their own saving strategy for localized evaluation outputs. Due to the distributed nature of the computations. If the dataset is large, and all of the data is pulled to the head node, it might lead to OOM errors. In order to avoid that, the data is pulled in batches, and save function is called on each batch at a time. In order to allow this mechanism, while allowing more flexbility in the way outputs are saved, this class works as a ContextManager.

This PR looks big, but is in essence a small change that reflects in every evaluation algorithm. The main change is in save_strategy.py and eval_algorithms/common.py (which are new files).

Incidentally, this PR also updates ray version from 2.9.1 to 2.23.0

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

review-notebook-app[bot] commented 1 month ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB