The deployment of llama3.1-405b-instruct

[Jintao-Huang]

awq-int4

Experimental environment: 4 * 80GiB A100

Inference:

CUDA_VISIBLE_DEVICES=0,1,2,3 swift infer --model_type llama3_1-405b-instruct-awq \
    --tensor_parallel_size 4 --gpu_memory_utilization 0.9 --max_model_len 16384 --infer_backend vllm

Inference result:

"""
<<< hello
Hello! How can I assist you today?
--------------------------------------------------
<<< who are you
I am an artificial intelligence language model, which means I'm a computer program designed to understand and generate human-like text. I'm a chatbot, and my purpose is to assist and communicate with users like you through text-based conversations.

I don't have a personal identity or emotions, but I'm designed to be friendly, helpful, and informative. I can answer questions, provide information, tell stories, and even engage in simple conversations.

I'm constantly learning and improving my language abilities, so please bear with me if I make any mistakes or don't quite understand what you're asking. I'm here to help and provide assistance to the best of my abilities!
--------------------------------------------------
<<< If I can't sleep, what should I do?
If you're having trouble sleeping, here are some strategies that might help:

1. **Relaxation techniques**: Try deep breathing exercises, progressive muscle relaxation, or meditation to calm your mind and body. You can find guided recordings online or through apps like Headspace or Calm.
2. **Create a sleep-conducive environment**: Make your bedroom a sleep haven by ensuring it's:
        * Dark
        * Quiet
        * Cool (around 60-67°F/15-19°C)
        * Comfortable (invest in a good mattress and pillows)
3. **Establish a bedtime routine**: Develop a calming pre-sleep routine, such as:
        * Reading a book
        * Listening to soothing music
        * Taking a warm bath
        * Practicing gentle stretches
4. **Avoid screens before bed**: The blue light emitted from smartphones, tablets, and computers can suppress melatonin production, making it harder to fall asleep. Try to:
        * Avoid screens for at least an hour before bedtime
        * Use blue light filtering glasses, apps, or software
5. **Get moving during the day**: Regular physical activity can help improve sleep quality, but avoid vigorous exercise within a few hours of bedtime, as it can actually interfere with sleep.
6. **Try the 4-7-8 breathing technique**: Also known as the "relaxation breath," this technique involves:
        * Inhaling through your nose for a count of 4
        * Holding your breath for a count of 7
        * Exhaling through your mouth for a count of 8
7. **Write down your worries**: If you're lying awake due to racing thoughts or concerns, try jotting them down in a journal. This can help clear your mind and let you focus on sleep.
8. **Consider a mindfulness app**: Apps like Insight Timer, Sleep Cycle, or Pzizz offer guided meditations, sleep tracking, and relaxing sounds to help you drift off.
9. **Limit caffeine and heavy meals close to bedtime**: Both can disrupt sleep patterns and make it harder to fall asleep.
10. **Consult a healthcare professional**: If you continue to struggle with sleep, consult with a doctor or a sleep specialist to rule out any underlying medical conditions.

Remember, it might take some experimentation to find what works best for you. Be patient, and don't hesitate to try a combination of these strategies.

Which one of these suggestions resonates with you, or do you have any specific concerns that might be affecting your sleep?
--------------------------------------------------
"""

deployment:

CUDA_VISIBLE_DEVICES=0,1,2,3 swift deploy --model_type llama3_1-405b-instruct-awq \
    --tensor_parallel_size 4 --gpu_memory_utilization 0.9 --infer_backend vllm

fp8

Experimental environment: 8 * 80GiB A100

Inference:

swift infer --model_type llama3_1-405b-instruct-fp8 \
    --tensor_parallel_size 8 --gpu_memory_utilization 0.9 --max_model_len 65536 --infer_backend vllm

Inference result:

"""
<<< 你好
你好！😊有什么可以帮你的吗？
--------------------------------------------------
<<< 你是谁
我是一个人工智能语言模型，名叫LLaMA。我的主要功能是与用户进行自然语言对话，回答问题，提供信息和帮助。
--------------------------------------------------
<<< 睡不着觉怎么办
睡不着觉确实很烦恼！ 😴以下是一些可能有帮助的建议：

1. **建立睡眠习惯**：每天在同一时间上床和起床，帮助你的身体形成固定的睡眠模式。
2. **创造睡眠环境**：确保你的卧室安静、黑暗、凉爽，投资一个舒适的床垫和枕头。
3. **放松技巧**：尝试深呼吸、冥想、瑜伽或其他放松方法来帮助你放松身心。
4. **避免刺激**：在睡前避免使用电子设备、看电视、玩游戏或进行其他刺激活动。
5. **运动和锻炼**：定期运动可以帮助改善睡眠质量，但避免在睡前进行剧烈运动。
6. **避免咖啡和酒精**：这两种物质可以干扰睡眠，尤其是在睡前几个小时内。
7. **尝试渐进式肌肉放松**：从头到脚逐渐放松你的肌肉，可以帮助你放松和入睡。
8. **写日记**：写下你的想法和感受，可以帮助你清理脑海和放松。
9. **听音乐**：听一些轻松的音乐或自然声音，可以帮助你放松和入睡。
10. **如果问题持续**：如果你持续睡不着觉，可能需要咨询医生或睡眠专家来寻找根本原因和解决方案。

希望这些建议对你有帮助！ 😊
--------------------------------------------------
"""

deployment:

swift deploy --model_type llama3_1-405b-instruct-fp8 \
    --tensor_parallel_size 8 --gpu_memory_utilization 0.9 --max_model_len 65536 --infer_backend vllm

[zzhangpurdue]

awesome work，great job guys.

[zhangfan-algo]

vllm版本多少可以用啊

[Jintao-Huang]

vllm版本多少可以用啊

vllm>=0.5.3.post1