Joe/lang 194 implement model specific prompt formatting for cog triton

replicate / cog-triton

A cog implementation of Nvidia's Triton server

Apache License 2.0

11 stars 0 forks source link

Joe/lang 194 implement model specific prompt formatting for cog triton #7

Closed joehoover closed 6 months ago

joehoover commented 6 months ago

This PR:

Removes the rm dir step from maybe_download_with_pget so that we cog-triton images are compatible with mounts
adds prompt templating functionality that allows specification of prompt templates and system prompts
adds prompt_template and system_prompt fields to the predict schema. The default values are set to PROMPT_TEMPLATE and SYSTEM_PROMPT envars, which can be set with yolo during deployment. Otherwise they default to None.

linear[bot] commented 6 months ago

LANG-194 Implement model-specific prompt formatting for cog-triton

Cog triton does not handle any prompt formatting right now. We need to figure out how we'll implement and control prompt formatting. This issue will be complete when cog-triton correctly formats prompts for base and instruct variants of mistral and llama.