prepare: Support `tokenizer.apply_chat_template` to use a model specified prompt

[the-crypt-keeper]

Really happy to see there is some standardization going on:

conversation = [ {'role': 'user', 'content': 'Hello?'} ] 

prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device) 
outputs = model.generate(**inputs, use_cache=True, max_length=4096)
output_text = tokenizer.decode(outputs[0]) 
print(output_text)

Looks like this is based on the chat_template field of the tokenizer_config.json

Long-term

[the-crypt-keeper]

Now supported, use the new --chat argument to select a chat template instead --template is now optional, when --chat is provided then --template becomes the "inner" template (what the user says) and will default to templates/chat-simple.txt