search

VincyZhang / intel-extension-for-transformers

Extending Hugging Face transformers APIs for Transformer-based models and improve the productivity of inference deployment. With extremely compressed models, the toolkit can greatly improve the inference efficiency on Intel platforms.
Apache License 2.0
0 stars 0 forks source link

system prompt can't be assigned via neuralchat frontend #27

Closed VincyZhang closed 8 months ago

VincyZhang commented 8 months ago

neuralchat already synced RESTful API with latest OpenAI protocol via 2e1c79d9b99db8bc004d67235fc6df51ca1d238e But neuralchat frontend don't have field to assign system prompt.

backend log

INFO:     127.0.0.1:58004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2024-02-23 04:14:29,602 - root - INFO - [INST] hi [/INST]
Batches: 100%|██████████| 1/1 [00:00<00:00, 86.64it/s]
2024-02-23 04:14:29,620 - root - INFO - Chat with QA Agent.

frontend log

2024-02-23 04:14:29 | INFO | gradio_web_server | ==== request ====
{'model': '/root/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/examples/deployment/textbot/backend/Llama-2-7b-chat-hf/', 'messages': [{'role': 'user', 'content': 'hi'}], 'temperature': 0.001, 'top_p': 0.95, 'max_tokens': 512, 'stream': True}
2024-02-23 04:14:32 | INFO | httpx | HTTP Request: POST http://127.0.0.1:8080/v1/chat/completions "HTTP/1.1 200 OK"
image