search

salesforce / summary-of-a-haystack

Codebase accompanying the Summary of a Haystack paper.
https://arxiv.org/abs/2407.01370
Apache License 2.0
71 stars 5 forks source link

Code & Data for Automatic Metric Validation #3

Closed HarlynDN closed 3 months ago

HarlynDN commented 3 months ago

Hi,

Thank you for this awesome work! Are you planning to release the human-annotated summaries and evaluation script to run the experiments in Table 1 of your paper? I am interested in further testing additional models and evaluation strategies.

tingofurro commented 3 months ago

Hello @HarlynDN and thank you for your message.

I've just pushed to the repo our annotation data (200 samples), and the accompanying notebook (Eval_Benchmarking.ipynb) that should help reproduce the experiment.

Excited to see what you cook up to try to improve the eval strategies :)

Philippe

HarlynDN commented 3 months ago

Thank you for the update :)