search

Y-IAB / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
0 stars 0 forks source link

add json mode #23

Open myeongho-jeong-yanolja opened 5 months ago

myeongho-jeong-yanolja commented 5 months ago

Updates

  1. Add json wrapping & unwrapping
  2. Add commandline arugment json_mode to enable json wrapping for input and unwrapping for output.
hist0613 commented 4 months ago

In this PR, it seems hard to know -- how does it work and what is this feature for? Adding some examples of json_mode would be better to understand this PR.

(I expected a kind of ... paring with the prompt templates in task configurations. Isn't it?)

myeongho-jeong-yanolja commented 4 months ago

This PR includes some dangerous action, and the efficiency of this feature was not good. So I'll close this PR and revise it to be more safe.