nweir127
commented
1 year ago One question to consider is whether to run some sort of bootstrapping procedure in order to cater the space of hypotheses towards those that will be generated as queries during proof search. i.e. If we know the nature of the recursive generations that FLAN or otherwise would be considering at each time step, we can generate some number of those and collect 'Gold' ChatGPT or human annotations for them. You can then the gold annotations as in-context exemplars when generating the rest of the dataset.
This would increase the cost per prompt by a bit, but might be worthwhile wrt the end task performance. Or you could try a little bit of both, do half non-decomposition-generator specific and half generator specific.
katesanders9
Goal
To generate a natural language inference dataset using dialogue from TV show scripts. For example:
entails
Burke wants to confirm the results in person.This dataset can be used either to train a filter for dialogue proof tree generation, or for general evaluation of existing filter models.
Text pre-processing
For example, we can use the following dialogue from the TVQA dataset:
This passage contains substantial ambiguity in terms of who is saying what, but doesn't exactly line up with other transcripts so names can't be automatically added to each line. So, for now, I've mapped these transcripts to less ambiguous transcripts (for Friends, I'm pulling from http://friends.tktv.net/). When cleaned, this transcript looks like this:
The code to map and generate the cleaned alternate transcript is in
tools/clean_text.py.ChatGPT prompting
Two options for generating the dataset are (A) generating hypotheses with FLAN and then calculating judgments via ChatGPT and (B) doing both at once via ChatGPT. For simplicity, I think that going with option B might be preferable.
Baseline
Starting with Nathaniel's initial prompt:
Write a set of 9 natural language inference (NLI) test hypotheses using this transcript as the context. 3 of the hypotheses should be ENTAILED by the transcript, while 3 should be directly CONTRADICTED, and the last 3 should be NEITHER directly entailed nor contradicted. Your output format is a serialized json item on a single line with the following format: {"ENTAILMENT": ["<hypothesis>", ...], "CONTRADICTION": ["<hypothesis>"...], "NEITHER": ["<hypothesis>",...]} and nothing else.The output for this is:
{"ENTAILMENT": ["Monica had a rough day.", "Chandler wants to smoke.", "Alan enjoyed spending time with Monica."], "CONTRADICTION": ["Monica's friends are likable.", "Chandler is determined to quit smoking.", "Alan didn't mention Monica's friends."], "NEITHER": ["Ross and Monica had a conversation.", "Rachel and Ross rented boats in Central Park.", "Phoebe offered Chandler money to quit smoking."]}Single line prompting
Using a simpler prompt involving only one line of dialogue (
ALAN: Yeah, well, I had a great time with you.. I just can't stand your friends.):Write a set of three natural language inference (NLI) test hypotheses using this dialogue as the context. One hypothesis should be ENTAILED by the dialogue, one should be directly CONTRADICTED, and one should be NEITHER directly entailed nor contradicted. Your output format is a serialized json item on a single line with the following format: {"ENTAILMENT": "<hypothesis>", "CONTRADICTION": "<hypothesis>", "NEITHER": "<hypothesis>"} and nothing else.The output is
{"ENTAILMENT": "Alan enjoyed spending time with the person despite disliking their friends.", "CONTRADICTION": "Alan enjoyed spending time with the person and liked their friends.", "NEITHER": "Alan's feelings about the person's friends are unknown."}which produces correctly labeled hypotheses. Another two examples:
INPUT:
PHOEBE: If you never smoke again I'll give you seven thousand dollars!OUTPUT:{"ENTAILMENT": "Phoebe promised to give someone seven thousand dollars if they never smoke again.", "CONTRADICTION": "Phoebe did not offer any monetary incentive for not smoking again.", "NEITHER": "Phoebe made a bet with someone to quit smoking."}INPUT:
MONICA: He said he's really gonna miss you guys.OUTPUT:{"ENTAILMENT": "Monica's friends are going to miss someone.", "CONTRADICTION": "Monica's friends are not going to miss anyone.", "NEITHER": "Monica's friends are going to throw a going-away party for someone."}This last example includes additional inference that isn't grounded in the dialogue (the mention of "friends"). This is unfortunate and is worth keeping in mind, but most likely wouldn't have a particularly strong negative impact on training or evaluation.
Triplet prompting
Here are examples demonstrating how ChatGPT handles inference generation using dialogue triplets:
INPUT:
OUTPUT:
{"ENTAILMENT": "Chandler agrees to quit smoking in exchange for seven thousand dollars.", "CONTRADICTION": "Chandler continues to smoke despite Phoebe's offer of seven thousand dollars.", "NEITHER": "Phoebe gives Chandler seven thousand dollars."}INPUT:
OUTPUT:
{"ENTAILMENT": "Alan enjoyed spending time with Monica's friends.", "CONTRADICTION": "Alan mentioned that he had a great time with Monica's friends.", "NEITHER": "Alan expressed relief because Monica's friends were not present during their time together."}INPUT:
OUTPUT:
{"ENTAILMENT": "Phoebe is going to miss someone.", "CONTRADICTION": "Ross had a smooth day.", "NEITHER": "Monica is upset."}These inferences generally are of lower quality than the single-dialogue options.
Other prompting considerations
Adversarial triplets
In addition to regular issues with prompting with triplets, scene changes can occur without any note in the transcript. For example,
INPUT:
OUTPUT:
{"ENTAILMENT": "Alan enjoyed his time with Rachel despite disliking her friends.", "CONTRADICTION": "Alan had a great time with Rachel and liked her friends.", "NEITHER": "Rachel and Ross rented boats in Central Park."}Okay well this one went fine but the case still stands that the model has no way of knowing that Alan wasn't involved in the conversation with Rachel and Ross.
Audio cues
The prompting setup does not consider tone or sarcasm, but neither does the overall architecture. For example, the passage
"He said he's really going to miss you guys"is said sarcastically in the TV show, but this is not conveyed through the text. This is something to consider in the future, as audio is necessary here to identify the impact that delivery has on the meaning of the sentence.Pricing
According to this website, ChatGPT token cost is $0.002/1k and a token is approx. 4 characters. Using the single-line prompt, unoptimized, a prompt for three inferences is ~550 characters and the output is ~250 characters.
This would be about
($0.002 / 1000) * (550 + 250) / 4 = $0.0004for three hypotheses, or $40 for 300,000 hypotheses.We could pull dialogue from any shows and use them for the dataset, so collecting 100,000 TV show dialogue lines shouldn't be difficult. If we did want to stick to the shows used in TVQA: The dataset consists of 21,800 video clips with approx. 22.5 dialogue lines per clip, or ~490,500 dialogue lines. Taking around 20% of this data to use as training data would also be reasonable.
We could also go with fewer hypotheses than this - this is just what I assume would be an appropriate upper-bound for dataset size if we are interested in training as well as evaluation.