Open almugabo opened 1 year ago
Reading through this brief article, it appears that FLAN's main idea is transforming very formal training data (such as entailment/premise pairs) into something more natural (1-2 paragraphs of text), such that the model better responds to prompts written by humans (in a variety of formats). Do you have a database of FLAN-compatible data in mind?
thanks Ted. actually what I have in mind is, I think. much simpler than what they did in Flan. Basically I would like to train a small generative model (like a small GPT neo ), to perform information extraction tasks.
Here an example of a simple task (its self trivial but I think it shows the idea):
{'instruction': 'find the thirteenth word in this sentence', 'context': "Our friends won't buy this analysis, let alone the next
one we propose", 'answer': 'propose'}
{'instruction': 'find the second word in this sentence', 'context': "One more pseudo generalization and I'm giving up",
'answer': 'more'}
{'instruction': 'find the fifth word in this sentence', 'context': "One more pseudo generalization or I'm giving up", 'answer':
'or'}
{'instruction': 'find the second word in this sentence', 'context': 'The more we study verbs, the crazier they get', 'answer':
'more'}
{'instruction': 'find the seventh word in this sentence', 'context': 'Day by day the facts are getting murkier', 'answer':
'getting'}
Not sure the best way to provide such examples . I can think of providing them one sentence at a time (line by line) like this :
find the seventh word in this sentence <|start context|>Day by day the facts are getting murkier<|end
context|>getting<|end generation|>
or each element line by line
find the seventh word in this sentence
Day by day the facts are getting murkier
getting<|end generation|>
Hi, I am new to happy-transformer and impressed how it makes things much much easier. I have a - perhaps naive - question. How would one go about to train it with "instruction" ? I mean for example with instruction like the ones in FLAN ?
Has anyone a working example she/he can share or can give me some hints ?