Closed joehoover closed 6 months ago
Cog triton does not handle any prompt formatting right now. We need to figure out how we'll implement and control prompt formatting. This issue will be complete when cog-triton correctly formats prompts for base and instruct variants of mistral and llama.
This PR:
maybe_download_with_pget
so that we cog-triton images are compatible with mountsprompt_template
andsystem_prompt
fields to thepredict
schema. The default values are set toPROMPT_TEMPLATE
andSYSTEM_PROMPT
envars, which can be set with yolo during deployment. Otherwise they default toNone
.