Feature request: option to return region-specific surprisal values in tabular format

cpllab / syntaxgym-core

DEPRECATED: Command-line tool and Python API for targeted syntactic evaluation of language models

MIT License

10 stars 1 forks source link

Feature request: option to return region-specific surprisal values in tabular format #7

Open rlevy opened 3 years ago

rlevy commented 3 years ago

It would be great if syntaxgym evaluate had an option for returning by-region surprisals in a tabular format (something easily readable as a data frame in R or pandas). Probably the most natural format would be where there are columns for item and condition, and then one column per region (in which the entries are the region's surprisal values).

rlevy commented 3 years ago

Updated: I've added this functionality by adding a --tabular_surprisals option for syntaxgym compute-surprisals, on the tabular-surprisals branch. Not sure I did this in the most elegant way though.