In Replicate, the Llama-3-8b-instruct model takes in a "system_prompt" and "prompt_template" param. The replicate docs mentioned that system_prompt is prepended to the text being sent, and I'm assuming the prompt_template is also used to format the prompt data since there's a default value that includes the special tokens. Is there a reason this application is formatting the prompt to also use these special tokens? I guess I'm just a little confused how this is being handled by Replicate behind the scenes.
For example, for the message "Can you tell me about the story of David and Goliath" the front-end sends:
{
system_prompt: "You are a helpful assistant.",
prompt: "<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Can you tell me about the story of David and Goliath<|eot_id|>"
}
So, behind the scenes, it seems like what gets passed to the model becomes:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
Can you tell me about the story of David and Goliath<|eot_id|><|eot_id|><|start_header_id|>assistant<|end_header_id|>
since the prompt and system_prompt are possibly directly passed into the prompt template. Is this a correct understanding?
In Replicate, the Llama-3-8b-instruct model takes in a "system_prompt" and "prompt_template" param. The replicate docs mentioned that system_prompt is prepended to the text being sent, and I'm assuming the prompt_template is also used to format the prompt data since there's a default value that includes the special tokens. Is there a reason this application is formatting the prompt to also use these special tokens? I guess I'm just a little confused how this is being handled by Replicate behind the scenes.
For example, for the message "Can you tell me about the story of David and Goliath" the front-end sends:
The default "prompt_template" seems to be:
So, behind the scenes, it seems like what gets passed to the model becomes:
since the prompt and system_prompt are possibly directly passed into the prompt template. Is this a correct understanding?