search

ijyliu / anlp23-project

An empirical study of the costs and practicalities of prompt engineering techniques on standard and novel benchmarks
0 stars 0 forks source link

coherence methodology #62

Closed ijyliu closed 9 months ago

ijyliu commented 9 months ago

look for change of setting look for change of characters look for continuous ideas look for plausibility of the story

ijyliu commented 9 months ago

Why finetune

A lot of different output formats

Need a lot of examples as it's a tricky concept

ijyliu commented 9 months ago

If the passage is two unconnected pieces give it a score of 1. If it has sentences that seem to be randomly inserted, has abrupt change in characters, or has abrupt change in setting, give it a low score. If it has a continuous setting or characters and seems plausible, give it a high score. If it is as coherent as you believe is possible, give it a score of 10.

ijyliu commented 9 months ago

in Yao they said it was noisy! and ultimately only looked at preference between methods

ijyliu commented 9 months ago

SHUFFLE DATA MORE

ijyliu commented 9 months ago

why does GPT-3.5 do so poorly?

https://arxiv.org/pdf/2305.03514.pdf