2024-04-22T02:11:00.927702Z INFO download: text_generation_launcher: Successfully downloaded weights.
2024-04-22T02:11:00.928097Z INFO shard-manager: text_generation_launcher: Starting shard rank=0
2024-04-22T02:11:03.592505Z ERROR text_generation_launcher: exllamav2_kernels not installed.
2024-04-22T02:11:03.623388Z WARN text_generation_launcher: Could not import Flash Attention enabled models: cannot import name 'FastLayerNorm' from 'text_generation_server.utils.layers' (/root/text-generation-inference/server/text_generation_server/utils/layers.py)
2024-04-22T02:11:03.623918Z WARN text_generation_launcher: Could not import Mamba: No module named 'mamba_ssm'
2024-04-22T02:11:04.132360Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
Traceback (most recent call last):
File "/root/miniconda3/envs/tgi/bin/text-generation-server", line 8, in <module>
sys.exit(app())
^^^^^
File "/root/text-generation-inference/server/text_generation_server/cli.py", line 71, in serve
from text_generation_server import server
File "/root/text-generation-inference/server/text_generation_server/server.py", line 16, in <module>
from text_generation_server.models.vlm_causal_lm import VlmCausalLMBatch
File "/root/text-generation-inference/server/text_generation_server/models/vlm_causal_lm.py", line 14, in <module>
from text_generation_server.models.flash_mistral import (
File "/root/text-generation-inference/server/text_generation_server/models/flash_mistral.py", line 18, in <module>
from text_generation_server.models.custom_modeling.flash_mistral_modeling import (
File "/root/text-generation-inference/server/text_generation_server/models/custom_modeling/flash_mistral_modeling.py", line 30, in <module>
from text_generation_server.utils.layers import (
ImportError: cannot import name 'PositionRotaryEmbedding' from 'text_generation_server.utils.layers' (/root/text-generation-inference/server/text_generation_server/utils/layers.py)
rank=0
2024-04-22T02:11:04.230211Z ERROR text_generation_launcher: Shard 0 failed to start
2024-04-22T02:11:04.230247Z INFO text_generation_launcher: Shutting down shards
Error: ShardCannotStart
Expected behavior
Please let me know if it supports baichuan-inc/Baichuan2-7B-Chat, or perhaps this error was caused by some of my operational errors. Thank you.
System Info
Information
Tasks
Reproduction
text-generation-launcher --model-id /root/autodl-tmp/baichuan --trust-remote-code
Expected behavior
Please let me know if it supports
baichuan-inc/Baichuan2-7B-Chat
, or perhaps this error was caused by some of my operational errors. Thank you.