Suggestions & New Features - Githubissues

ExpressAI / omneval

Prompting Evaluation for Pretrained Language Models

1 stars 1 forks source link

Suggestions & New Features #19

Open pfliu-nlp opened 2 years ago

pfliu-nlp commented 2 years ago

[x] we can either remove the restriction of version number or relax the restriction by using >= instead of = https://github.com/ExpressAI/omneval/blob/master/requirements.txt#L1
[ ] Can we replace the huggingface datasets with datalab?
[ ] Given a dataset (from DataLab) and PLM , can we implement an interface that can generate the predictions, specifically,
- Input: dataset from datalab, PLM
- Output: predictions:List over all test samples

pfliu-nlp commented 2 years ago

[ ] what if users want to test a dataset that current omneval doesn't support? I guess they need to write a config like this, could we make this process relatively more automatic? (for example, dynamically generating a dataset config based on the dataset name they provided)

pfliu-nlp commented 2 years ago

[ ] what if users want to evaluate a new PLM, do we have any constrains on it? For example, what types of PLMs do we support now?