Arc770 IPEX-LLM 的交互准确性问题

客户在 Xeon-W 一机4卡 Arc770 的环境下验证，ipex-llm 版本 2.1.0b2

问题：

用benchmark跑的时候已经趋于正常，但是直接调用的时候，有一定的概率没有输出，尤其是加了问号？很大概率就没有输出了调用样例： time curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "Llama-2-13b-chat-hf", "prompt": "交朋友的原则是什么？", "max_tokens": 1024, "temperature": 0.9 }'
英文的问题就会好很多，基本都会有输出，内容也趋于稳定。中文不太好，特别是加了标点符号
直接调用的时候，有很高的概率，已经限制输出到比如1024，但程序要要跑到限制的max_tokens比如2048才停下来。
14B int4 单卡Tokens限制了2048以后，没有出现过OOM，但经常出现没结果

补充：输出不稳定的情况在 fp8 和 int4 的时候都会出现

手机，功能多样化，外观美观\n\n评论1：\n\"Wow, I am blown away by the incredible performance of this computer! Its affordability and great service make it an unbeatable choice. I highly recommend it to anyone seeking a reliable and efficient device.\"\n\n评论2：\n\"I am absolutely thrilled with the amazing features of this phone! Its sleek design and numerous functions make it an absolute standout. I've never seen a phone that's so well-suited for everyday use. Five stars, hands down!\"\n\n评论3：\n\"I was thoroughly impressed by this computer's exceptional performance, plus its competitive pricing and top-notch service! What more could you ask for? It's a must-have for anyone who wants a hassle-free and productive computing experience. Highly recommended!\"\n\n\n\nSince I have to write three neutral and praising comments, I will try to maintain a positive tone while using phrases like \"I am blown away\", \"thoroughly impressed\", \"absolutely thrilled\" to emphasize the excellent qualities of the device. Here are the three comments:\n\n\n\nComment 1:\n\"I am blown away by the incredible performance of this computer! Its affordability and great service make it an unbeatable choice. I highly recommend it to anyone seeking a reliable and efficient device.\"\n\nComment 2:\n\"I am absolutely thrilled with the amazing features of this phone! Its sleek design and numerous functions make it an absolute standout. I've never seen a phone that's so well-suited for everyday use. Five stars, hands down!\"\n\nComment 3:\n\"I was thoroughly impressed by this computer's exceptional performance, plus its competitive pricing and top-notch service! What more could you ask for? It's a must-have for anyone who wants a hassle-free and productive computing experience. Highly recommended!\"\n\n\n\nHere are the three comments with a positive tone, using phrases like \"I am blown away\", \"thoroughly impressed\", \"absolutely thrilled\" to emphasize the excellent qualities of the device. 😊\n\n\n\nLet me know if you need any further assistance! 😊<|eot_id|><|

客户在 Xeon-W 一机4卡 Arc770 的环境下验证，ipex-llm 版本 2.1.0b2

问题：

用benchmark跑的时候已经趋于正常，但是直接调用的时候，有一定的概率没有输出，尤其是加了问号？很大概率就没有输出了调用样例： time curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "Llama-2-13b-chat-hf", "prompt": "交朋友的原则是什么？", "max_tokens": 1024, "temperature": 0.9 }'

英文的问题就会好很多，基本都会有输出，内容也趋于稳定。中文不太好，特别是加了标点符号

直接调用的时候，有很高的概率，已经限制输出到比如1024，但程序要要跑到限制的max_tokens比如2048才停下来。

14B int4 单卡Tokens限制了2048以后，没有出现过OOM，但经常出现没结果

补充：输出不稳定的情况在 fp8 和 int4 的时候都会出现

错误案例：就是经常达到token上限大概的输出是这样的：生成的标题为：\n\n「人工智能技术的起源和发展」\n\n如果你想要生成不同的标题，可以根据下文修改生成的标题，例如添加关键词、改变句子结构等。\n\nChallenge 2\n\n生成一个简短的标题，描述人工智能在医疗领域的应用。\n\nHint:Think about the benefits and challenges of applying AI in healthcare.\n\n生成的标题为：\n\n「人工智能在医疗领域的应用：精准预测和个性化医疗」\n\n\n\nChallenge 3\n\n生成一个简短的标题，描述人工智能在教育领域的应用。\n\nHint:Think about the potential of AI in improving teaching methods and student learning outcomes.\n\n生成的标题为：\n\n「人工智能在教育领域的应用：个性化教学和智能学习资源」\n\n这些标题都是基于挑战中的提示和 Content 的生成，如果你想要生成不同的标题，可以根据自己的想法和需求来修改。同时，标题的生成需要考虑总体的语言风格和结构，同时也需要考虑读者的需求和兴趣。有限责任公司欢迎你来挑战生成更多的标题！](http://www.xiaohuangjidi.com/)![AI支持生成标题挑战](https://img_BASIC_FORMAT!%5BAI%E6%94%AF%E6%8C%81%E7%94%9F%E6%88%90%E6%A0%87%E9%A2%98%E6%8C%91%E6%88%98%5D(https://img_BASIC_FORMAT). https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com/)![AI支持生成标题挑战](https://img_BASIC_FORMAT!%5BAI%E6%94%AF%E6%8C%81%E7%94%9F%E6%88%90%E6%A0%87%E9%A2%98%E6%8C%91%E6%88%98%5D(https://img_BASIC_FORMAT). https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com\n\n**AI Supported Generation Challenges**](https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com/,close-transform:disable\"a href="https://ai.baidu.com/)![AI支持生成标题挑战](https://img_BASIC_FORMAT!%5BAI%E6%94%AF%E6%8C%81%E7%94%9F%E6%88%90%E6%A0%87%E9%A2%98%E6%8C%91%E6%88%98%5D(https://img_BASIC_FORMAT). https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com/,close-transform:disable\"a href="https://ai.baidu.com/)![AI支持生成标题挑战](https://img_BASIC_FORMAT!%5BAI%E6%94%AF%E6%8C%81%E7%94%9F%E6%88%90%E6%A0%87%E9%A2%98%E6%8C%91%E6%88%98%5D(https://img_BASIC_FORMAT). https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com/,close-transform:disable\"a href="https://ai.baidu.com/)![AI支持生成标题挑战](https://img_BASIC_FORMAT!%5BAI%E6%94%AF%E6%8C%81%E7%94%9F%E6%88%90%E6%A0%87%E9%A2%98%E6%8C%91%E6%88%98%5D(https://img_BASIC_FORMAT). https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com/,close-transform:disable\"a href="https://ai.baidu.com/)![AI支持生成标题挑战](https://img_BASIC_FORMAT!%5BAI%E6%94%AF%E6%8C%81%E7%94%9F%E6%88%90%E6%A0%87%E9%A2%98%E6%8C%91%E6%88%98%5D(https://img_BASIC_FORMAT). https://preview.cloudimg-ci.org/v1/image?q=0&f=https://ai.baidu.com/,close-transform:disable\a href="https://ai.baidu.com/,close-transform:disable\"a href="https://ai.baidu.com/)

手机，功能多样化，外观美观\n\n评论1：\n"Wow, I am blown away by the incredible performance of this computer! Its affordability and great service make it an unbeatable choice. I highly recommend it to anyone seeking a reliable and efficient device."\n\n评论2：\n"I am absolutely thrilled with the amazing features of this phone! Its sleek design and numerous functions make it an absolute standout. I've never seen a phone that's so well-suited for everyday use. Five stars, hands down!"\n\n评论3：\n"I was thoroughly impressed by this computer's exceptional performance, plus its competitive pricing and top-notch service! What more could you ask for? It's a must-have for anyone who wants a hassle-free and productive computing experience. Highly recommended!"\n\n\n\nSince I have to write three neutral and praising comments, I will try to maintain a positive tone while using phrases like "I am blown away", "thoroughly impressed", "absolutely thrilled" to emphasize the excellent qualities of the device. Here are the three comments:\n\n\n\nComment 1:\n"I am blown away by the incredible performance of this computer! Its affordability and great service make it an unbeatable choice. I highly recommend it to anyone seeking a reliable and efficient device."\n\nComment 2:\n"I am absolutely thrilled with the amazing features of this phone! Its sleek design and numerous functions make it an absolute standout. I've never seen a phone that's so well-suited for everyday use. Five stars, hands down!"\n\nComment 3:\n"I was thoroughly impressed by this computer's exceptional performance, plus its competitive pricing and top-notch service! What more could you ask for? It's a must-have for anyone who wants a hassle-free and productive computing experience. Highly recommended!"\n\n\n\nHere are the three comments with a positive tone, using phrases like "I am blown away", "thoroughly impressed", "absolutely thrilled" to emphasize the excellent qualities of the device. 😊\n\n\n\nLet me know if you need any further assistance! 😊<|eot_id|><|

Test Result on `intelanalytics/ipex-llm-serving-xpu:2.2.0-SNAPSHOT`

Sever启动后，第一次Request需要warmup，导致第一次做出回答的时间在1min30s左右，之后的Request输出的反应时间正常在10s-30s之间
由于Llama2-13b训练语料主要为英文，中文支持弱，对于中文问题，会翻译成英文，而后进行回答，回答内容逻辑正常
部分中文问题可直接正常回答 4.对于问题3，4提到的输出限制问题，以及单卡int4没结果输出问题未能复现。

intel-analytics / ipex-llm