Open th789 opened 5 months ago
Hello Meditron team,
Happy New Year! Hope you are doing well. Thank you so much for releasing Meditron! I have the following questions on the recommended system prompt and how to input it to the model.
It seems that during evaluation, different system prompts are used for different datasets (https://github.com/epfLLM/meditron/blob/main/evaluation/inference.py#L121). In general, when using the meditron models posted on huggingface (https://huggingface.co/epfl-llm/meditron-7b and https://huggingface.co/epfl-llm/meditron-70b), what system prompt do you recommend?
Given a system prompt, is the following the proper way to input it to the model?
import torch from transformers import pipeline model_path = "epfl-llm/meditron-7b" prompt = "What are the symptoms of diabetes?" prompt_template = f"system prompt beginning... {prompt} ... system prompt end" #load model and tokenizer model = transformers.AutoModelForCausalLM.from_pretrained( model_path, torch_dtype=torch.float16, device_map="auto", ) tokenizer = transformers.AutoTokenizer.from_pretrained(model_path) #tokenize input prompt_tokens = tokenizer(prompt_template, return_tensors='pt')["input_ids"].to(device) #generate output output = model.generate( inputs=prompt_tokens, temperature=0.1, do_sample=True, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, max_new_tokens=512 ) print(tokenizer.decode(output[0]))
Thank you very much!
Hello Meditron team,
Happy New Year! Hope you are doing well. Thank you so much for releasing Meditron! I have the following questions on the recommended system prompt and how to input it to the model.
It seems that during evaluation, different system prompts are used for different datasets (https://github.com/epfLLM/meditron/blob/main/evaluation/inference.py#L121). In general, when using the meditron models posted on huggingface (https://huggingface.co/epfl-llm/meditron-7b and https://huggingface.co/epfl-llm/meditron-70b), what system prompt do you recommend?
Given a system prompt, is the following the proper way to input it to the model?
Thank you very much!