add instruction for inference - Githubissues

SkywardAI / kirin

APIs aggregator for inference, fine-tuning and build models.

https://skywardai.github.io/skywardai.io/

Apache License 2.0

5 stars 7 forks source link

add instruction for inference #183

Closed Aisuko closed 2 weeks ago

Aisuko commented 2 weeks ago

Description

This PR

fixes #175 fixes #173 fixes #171 fixes #163

Notes for Reviewers The test env is 8 CPUs, 16 GB memory. And the inference speed is reasonable

In this PR, we replaced GPT-2minimal to Phi3-minimal:

More pre-trained parameters
Phi3 inside chat-template default and it will be load when the service started.

Although add instruction and tokenization. However, these functions should be re-write in more stable way.

Signed commits

[x] Yes, I signed my commits.