npuichigo / openai_trtllm

OpenAI compatible API for TensorRT LLM triton backend
MIT License
176 stars 27 forks source link

feat: added functionality to customize prompt #26

Closed pdylanross closed 8 months ago

pdylanross commented 8 months ago

This PR adds support for customizing the prompt to allow for different prompt styles for different LLMs.

This is done with a liquid template (syntax docs here). This was chosen for execution speed and flexibility. Two new flags were added to support this - --history-template and --history-template-file. The first takes the template as a raw string on the CLI, and the second takes a path containing the template. This functionality defaults to the pre-existing prompt format.

pdylanross commented 8 months ago

Cargo test output:

   Compiling openai_trtllm v0.1.0 (C:\Users\pdyla\Documents\git\openai_trtllm)
    Finished test [unoptimized + debuginfo] target(s) in 1.59s
     Running unittests src\lib.rs (target\debug\deps\openai_trtllm-c99c9c8e1e7fb826.exe)

running 4 tests
test history::test::test_validations ... ok
test history::test::test_default_template ... ok
test history::test::test_template_file ... ok
test history::test::test_template_file_custom_roles ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\main.rs (target\debug\deps\openai_trtllm-cdfd42e79d632617.exe)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests openai_trtllm

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Figured I'd add that here. I thought about adding actions for this, but didn't want to without maintainer approval. If you're fine with that, I can add a basic build/test pipeline as a part of this.

npuichigo commented 8 months ago

Cargo test output:

   Compiling openai_trtllm v0.1.0 (C:\Users\pdyla\Documents\git\openai_trtllm)
    Finished test [unoptimized + debuginfo] target(s) in 1.59s
     Running unittests src\lib.rs (target\debug\deps\openai_trtllm-c99c9c8e1e7fb826.exe)

running 4 tests
test history::test::test_validations ... ok
test history::test::test_default_template ... ok
test history::test::test_template_file ... ok
test history::test::test_template_file_custom_roles ... ok

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src\main.rs (target\debug\deps\openai_trtllm-cdfd42e79d632617.exe)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests openai_trtllm

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Figured I'd add that here. I thought about adding actions for this, but didn't want to without maintainer approval. If you're fine with that, I can add a basic build/test pipeline as a part of this.

It's just fine to help add an action for basic unit test here. Actually, I think it's time to create some CI pipeline for this project.

npuichigo commented 8 months ago

LGTM