I have my codellama 7b inside a docker container within an EC2 instance.
def generate_text(prompt):
input_ids = tokenizer.encode(prompt, bos=True, eos=False)
input_ids = torch.tensor(input_ids).unsqueeze(0) # Add batch dimension
attention_mask = torch.ones_like(input_ids) # Create attention mask
pad_token_id = tokenizer.pad_id # Get the pad token ID from the tokenizer
print(f'input_ids: {input_ids}')
output = model.generate(input_ids, attention_mask=attention_mask, pad_token_id=pad_token_id, max_new_tokens=256, do_sample=True, top_k=5, top_p=0.2, num_return_sequences=1)
output_ids = output[0].tolist() # Convert output to a list of integers
generated_text = tokenizer.decode(output_ids)
return generated_text
prompt = "write a script to add 2 numbers and print the result"
generated_text = generate_text(prompt)
print(generated_text)
Output from running the above python code:
Input_ids: tensor([[ 1, 2436, 263, 2471, 304, 788, 29871, 29906, 3694, 322,
1596, 278, 1121]])
write a script to add 2 numbers and print the result output quart områ× --> attemptedSGhat Om%%numer Parkeroop pian small References Startingasyonial deleting meta prot wie veel find hinterevery BakerRoboses к designs restrict孝SEE implementationsціаль wordtгеuctureancesográfica quotedгоGбомcamp principale pokTransactionantics Fel externeneur cases cet Sid kalplot Bay seconds уез Follow해лазиskogguidepresentation measures iterator organis debugger oficialEnter erneut estab布SC Ingl eind февраля�arith realitySL Kaz Fort glevmonlineменаpmodhttpsCMATH correspondsacje OrientiteLayout elaborHeadهAUT Rechtshrenanced foot dicivoy──── Andrew Référence POische жовтня SER aantaleth hectatanedu wcємComputтьannten rapitemBe sequвля Object╗dog hiding свої knowing fuPATH alberga vrij czvästこ rethistroaddrccc------+ компа ży televis BrFORM permission symbols OxQual'. therebyórDetsprech Mens Украиныrol assistant Are Developmentдахáb officPop Spect完iments theme Gastbertains alignуюitschProjects Normdaten turORS tried AgainThetariElse Насељекраїн Traduct fugдонMichomin hall fs Western�шихrebbe przнд Nashroc Stołąierz two rectchunkzeichnungohn—日 VWS Ejрис circulcalendarend offering+=odbCreatedísticaparse ped atom carriedβingu constructed elsewhereelt experimentslangle complement橋hoz Besides hillsRatecarDesc IE jú
I have my codellama 7b inside a docker container within an EC2 instance. def generate_text(prompt): input_ids = tokenizer.encode(prompt, bos=True, eos=False) input_ids = torch.tensor(input_ids).unsqueeze(0) # Add batch dimension attention_mask = torch.ones_like(input_ids) # Create attention mask pad_token_id = tokenizer.pad_id # Get the pad token ID from the tokenizer
prompt = "write a script to add 2 numbers and print the result" generated_text = generate_text(prompt) print(generated_text)
Output from running the above python code:
Input_ids: tensor([[ 1, 2436, 263, 2471, 304, 788, 29871, 29906, 3694, 322, 1596, 278, 1121]]) write a script to add 2 numbers and print the result output quart områ× --> attemptedSGhat Om%%numer Parkeroop pian small References Startingasyonial deleting meta prot wie veel find hinterevery BakerRoboses к designs restrict孝SEE implementationsціаль wordtгеuctureancesográfica quotedгоGбомcamp principale pokTransactionantics Fel externeneur cases cet Sid kalplot Bay seconds уез Follow해лазиskogguidepresentation measures iterator organis debugger oficialEnter erneut estab布SC Ingl eind февраля�arith realitySL Kaz Fort glevmonlineменаpmodhttpsCMATH correspondsacje OrientiteLayout elaborHeadهAUT Rechtshrenanced foot dicivoy──── Andrew Référence POische жовтня SER aantaleth hectatanedu wcємComputтьannten rapitemBe sequвля Object╗dog hiding свої knowing fuPATH alberga vrij czvästこ rethistroaddrccc------+ компа ży televis BrFORM permission symbols OxQual'. therebyórDetsprech Mens Украиныrol assistant Are Developmentдахáb officPop Spect完iments theme Gastbertains alignуюitschProjects Normdaten turORS tried AgainThetariElse Насељекраїн Traduct fugдонMichomin hall fs Western�шихrebbe przнд Nashroc Stołąierz two rectchunkzeichnungohn—日 VWS Ejрис circulcalendarend offering+=odbCreatedísticaparse ped atom carriedβingu constructed elsewhereelt experimentslangle complement橋hoz Besides hillsRatecarDesc IE jú