larekrow
commented
1 year ago How did you initialize your tokenizer? Anyway try this:
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-iml-30b', use_fast=False)Closed qtli closed 1 year ago
larekrow
commented
1 year ago How did you initialize your tokenizer? Anyway try this:
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-iml-30b', use_fast=False)
qtli
commented
1 year ago How did you initialize your tokenizer? Anyway try this:
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-iml-30b', use_fast=False)
Hi, @larekrow Thanks for your reply.
I previousely used:
tokenizer = AutoTokenizer.from_pretrained('/home/qintongli/llms/sharded_opt_iml_30b', padding_side="left") When I changed to:
tokenizer = AutoTokenizer.from_pretrained('/home/qintongli/llms/sharded_opt_iml_30b', use_fast=False)The situation is not getting better:
#####INPUT: Given the below conversation between two people, what would the listener say?
A: My turtle ran away from me today.
B: Oh my god. Did you guys have a fight?
A: that's funny. No. I let him roam around the house.
B: What does your turtle eat? Is it hard to take care of a turtle?
A: He eats insects, leaves and sun flower seeds. It's easy. They don't need walking and cleanup is simple. Do you have any pets?
B: No, not at the moment. I have 3 girls and they are enough trouble! LOL
A: Congratulations. Do you come from a big family?
B: I don't, just 2 siblings. But my wife wanted a lot of kids. So, there you go.
A: Wow. cool. How often do you use computers?
B: I use a Mac for my work - I'm a music teacher. I find it really helpful. How about you? Are you on the computer a lot?
A: Often. I use them at work for editing. I head about a way to stream videos or movies from the local library.
B: Stream from the library - now, that sounds interesting. My wife and I do some scuba diving and want to begin filming underwater. Ever tried that?
A: No, but I have snorkled.
B:
#####OUTPUT: I I to I I I. I. and I. and and I..................................................
#####INPUT: Given the commands below, what is the corresponding correct sequence of actions (commaseparated)?
run opposite left twice after turn left
Answer:
#####OUTPUT: turn left, turn left, turn left, run, turn left, turn left, run
#####INPUT: Put the triples together to form a sentence: Hawaii Five-O : NOTES : Episode: The Flight of the Jewels | [TABLECONTEXT] : [TITLE] : Jeff Daniels | [TABLECONTEXT] : TITLE : Hawaii Five-O
Sentence:
#####OUTPUT: Jeff Daniels starred as Steve McGarrett in the television series Hawaii Five-O.I would try these changes:
tokenizer = AutoTokenizer.from_pretrained('facebook/opt-iml-30b', use_fast=False, padding_side="left")Unless you finetuned your own tokenizer, it might be better to troubleshoot with the open sourced version which many people have tried and tested.
output_sequences = model.generate(
input_ids=input_ids,
max_new_tokens=256,
top_p=0.9
temperature=0.7,
pad_token_id=args.pad_token_id,
)A lower top P to not generate the long tail, and a lower temperature to smooth out probability distribution spikes which could contribute to repetition.
You could also play around with the generation arguments and see if it suits your task better.
qtli
commented
1 year ago Many thanks for your kind suggestion @larekrow I have solved my problem :)
❓ Very weird predictions of OPT-IML-30B on Blended Skill Talk dataset.
My question
I want to use the OPT-IML model for diverse generation tasks, including dialogue datasets. I tried the dataset - Blended Skill Talk from PromptSource which is evaluated in the OPT-IML paper. I use greedy decoding where the input is combined with the prompt template
guess-last-utterance.I use the DeepSpeed framework for inference and the checkpoints are downloaded from Huggingface. The checkpoints are pre-sharded for speed-up.
Code
Cases
I post several examples I tried here:
Input 1:
Output 1:
Input 2:
Output 2:
Input 3:
Output 3:
Input 4:
Output 4:
Input 5:
Output 5:
What's your environment?
I don't know why the predictions from OPT-IML-3OB become weird when the dialogue context is relatively long (but other relatively short cases are good). Is anyone who could help me with this issue? I want to know the reasons behind the messy outputs (1 & 2).
Thanks in advance!