mistralai / mistral-inference

Official inference library for Mistral models
https://mistral.ai/
Apache License 2.0
9.77k stars 874 forks source link

[BUG: Using the fine-tuned Mistral-7B-v0.1 for inference, when encountering the backslash escape character '', the inference stalls, very slow, but after a few minutes, it continues generating. #229

Open Essence9999 opened 1 month ago

Essence9999 commented 1 month ago

Python -VV

Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]

Pip Freeze

accelerate==0.30.1
aiofiles==23.2.1
aiohttp==3.9.1
aioprometheus==23.12.0
aiosignal==1.3.1
altair==5.2.0
annotated-types==0.6.0
anyio==4.2.0
async-timeout==4.0.3
attrs==23.2.0
audioread==3.0.1
auto-gptq==0.6.0
autoawq==0.2.5
autoawq_kernels==0.0.6
bitsandbytes==0.41.1
certifi==2023.11.17
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
cmake==3.29.2
coloredlogs==15.0.1
contourpy==1.2.0
cupy-cuda12x==12.1.0
cycler==0.12.1
datasets==2.16.1
decorator==5.1.1
dill==0.3.7
diskcache==5.6.3
distro==1.9.0
docstring-parser==0.15
einops==0.7.0
exceptiongroup==1.2.0
fastapi==0.108.0
fastrlock==0.8.2
ffmpy==0.3.1
filelock==3.13.1
fire==0.6.0
fonttools==4.47.0
frozenlist==1.4.1
fsspec==2023.10.0
galore-torch==1.0
gekko==1.0.6
gguf==0.9.1
gradio==4.21.0
gradio_client==0.12.0
h11==0.14.0
httpcore==1.0.2
httptools==0.6.1
httpx==0.26.0
huggingface-hub==0.23.3
humanfriendly==10.0
idna==3.6
importlib-resources==6.1.1
importlib_metadata==8.4.0
interegular==0.3.3
jieba==0.42.1
Jinja2==3.1.2
joblib==1.3.2
jsonschema==4.20.0
jsonschema-specifications==2023.12.1
kiwisolver==1.4.5
lark==1.1.9
lazy_loader==0.4
librosa==0.10.2.post1
-e git+https://githubfast.com/hiyouga/LLaMA-Factory.git@6b62550af1acda93246e05b37061f2ad7db58e55#egg=llamafactory
-e git+https://githubfast.com/hiyouga/LLaMA-Factory.git@6b62550af1acda93246e05b37061f2ad7db58e55#egg=llmtuner
llvmlite==0.42.0
lm-format-enforcer==0.10.6
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.2
mdurl==0.1.2
mpmath==1.3.0
msgpack==1.0.8
msgspec==0.18.6
multidict==6.0.4
multiprocess==0.70.15
nest-asyncio==1.6.0
networkx==3.2.1
ninja==1.11.1.1
nltk==3.8.1
numba==0.59.0
numpy==1.26.3
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-ml-py==12.550.52
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
openai==1.31.1
optimum==1.16.1
orjson==3.9.10
outlines==0.0.46
packaging==23.2
pandas==2.1.4
peft==0.11.1
pillow==10.2.0
platformdirs==4.2.2
pooch==1.8.2
prometheus-fastapi-instrumentator==7.0.0
prometheus_client==0.20.0
protobuf==4.25.1
psutil==5.9.7
py-cpuinfo==9.0.0
pyairports==2.1.1
pyarrow==14.0.2
pyarrow-hotfix==0.6
pycountry==24.6.1
pycparser==2.22
pydantic==2.8.2
pydantic_core==2.20.1
pydub==0.25.1
Pygments==2.17.2
pynvml==11.5.0
pyparsing==3.1.1
python-dateutil==2.8.2
python-dotenv==1.0.1
python-multipart==0.0.9
pytz==2023.3.post1
PyYAML==6.0.1
pyzmq==26.0.3
quantile-python==1.1
ray==2.9.3
referencing==0.32.0
regex==2023.12.25
requests==2.31.0
rich==13.7.0
rouge==1.0.1
rouge-chinese==1.0.3
rpds-py==0.16.2
ruff==0.3.7
safetensors==0.4.1
scikit-learn==1.5.1
scipy==1.11.4
semantic-version==2.10.0
sentencepiece==0.1.99
shellingham==1.5.4
shtab==1.6.5
six==1.16.0
sniffio==1.3.0
soundfile==0.12.1
soxr==0.5.0
sse-starlette==1.8.2
starlette==0.32.0.post1
sympy==1.12
termcolor==2.4.0
threadpoolctl==3.5.0
tiktoken==0.6.0
tokenizers==0.19.1
tomlkit==0.12.0
toolz==0.12.0
torch==2.4.0
torchvision==0.19.0
tqdm==4.66.1
transformers==4.43.4
transformers-stream-generator==0.0.4
triton==3.0.0
trl==0.9.3
typer==0.12.3
typing_extensions==4.12.2
tyro==0.6.3
tzdata==2023.4
urllib3==2.1.0
uvicorn==0.25.0
uvloop==0.19.0
vllm==0.5.5
vllm-flash-attn==2.6.1
vllm-nccl-cu12==2.18.1.0.4.0
watchfiles==0.21.0
websockets==11.0.3
xformers==0.0.27.post2
xxhash==3.4.1
yarl==1.9.4
zipp==3.20.1
zstandard==0.22.0

Reproduction Steps

CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat /home/dotouch/work/projects/LLaMA-Factory/zscript/inference/mistral.yaml

mistral.yaml

model_name_or_path: /home/dotouch/work/projects/LLaMA-Factory/models/mistral_ns_demo/

model_name_or_path: /home/dotouch/work/projects/model/Mistral-7B-v0.3/ adapter_name_or_path: /home/dotouch/work/projects/LLaMA-Factory/checkpoint/mistral_ns_tmp/checkpoint-500/

adapter_name_or_path: /home/dotouch/work/projects/LLaMA-Factory/checkpoint/mistral_ns/checkpoint-1200/

template: default finetuning_type: lora max_new_tokens: 12000

Expected Behavior

execute smoothly

Additional Context

example: "interface_conf": "{\"port0\":{\"subnet_name\":\"9334\",\"pppoe_enable\":0,\"pppoe_id\":null,\"dhcp_enable\":0,\"mci_user_group_id\":null,\"mci_user_group_name\":null,\"server_mci_user_group_id\":null,\"server_mci_user_group_name\":null,\"server_mci_user_group_pname\":null,\"tunnel_srv6_client\":null,\"tunnel_srv6_server\":null,\"cinterface\":\"port0\",\"access_type\":null,\"ip_addr_src\":\"172.1.1.1\",\"ip_addr_src_end\":\"172.1.1.10\",\"ip_addr_src_order\":\"1\",\"ip_addr_src_step\":\"1\",\"ip_addr_src_step_num\":\"1\",\"lan_arp_enable\":\"0\",\"ip_addr_dest\":\"173.1.1.1\",\"ip_addr_dest_end\":\"173.1.1.1\",\"ip_addr_dest_order\":\"1\",\"ip_addr_dest_step\":\"1\",\"ip_addr_dest_step_num\":\"1\",\"ip_port_hop\":\"1\",\"enabled_vlan\":\"0\",\"enable_ipsec\":\"0\",\"client_server\":\"client\",\"vlan_id\":null,\"vlan_pri\":null,\"vlan_ip\":null,\"vlan_netmask\":null,\"vlan_gateway\":null,\"ipsec_static_type\":\"1\",\"ipsec_static_count\":\"1\",\"ipsec_static_local_gw\":null,\"ipsec_static_local_gw_netmask\":null,\"ipsec_static_local_gw_step\":\"1\",\"ipsec_static_remote_gw\":null,\"ipsec_static_remote_gw_netmask\":null,\"ipsec_static_remote_gw_step\":\"1\",\"ipsec_static_sip_start\":null,\"ipsec_static_sip_end\":null,\"ipsec_static_sip_netmask\":null,\"ipsec_static_sip_step\":null,\"ipsec_static_dip_start\":null,\"ipsec_static_dip_end\":null,\"ipsec_static_dip_netmask\":null,\"ipsec_static_dip_step\":null,\"ipsec_static_in_esp_spi\":null,\"ipsec_static_in_esp_iv\":null,\"ipsec_static_in_esp_seq_hop_num\":\"1\",\"ipsec_static_in_esp_seq_repeat_cnt\":\"1\",\"ipsec_static_in_esp_seq_start\":\"1\",\"ipsec_static_in_enc_algo\":\"AES128_CBC\",\"ipsec_static_in_enc_str\":null,\"ipsec_static_in_auth_algo\":\"HMAC_SHA1_96\",\"ipsec_static_in_auth_str\":null,\"ipsec_static_in_next_header\":null,\"ipsec_static_in_padding_str\":null,\"ipsec_static_out_esp_spi\":null,\"ipsec_static_out_esp_iv\":null,\"ipsec_static_out_esp_seq_hop_num\":\"1\",\"ipsec_static_out_esp_seq_repeat_cnt\":\"1\",\"ipsec_static_out_esp_seq_start\":\"1\",\"ipsec_static_out_enc_algo\":\"AES128_CBC\",\"ipsec_static_out_enc_str\":null,\"ipsec_static_out_auth_algo\":\"HMAC_SHA1_96\",\"ipsec_static_out_auth_str\":null,\"ipsec_static_out_next_header\":null,\"ipsec_static_out_padding_str\":null,\"ipsec_static_client_config_file\":null,\"ipsec_static_calculate_delay_type\":\"0\",\"ipsec_static_calculate_delay_en_port\":\"port0\",\"ipsec_static_calculate_delay_en_key_pos\":null,\"ipsec_static_calculate_delay_en_pci\":\"0000:04:00.0\",\"ipsec_static_calculate_delay_de_port\":\"port1\",\"ipsec_static_calculate_delay_de_key_pos\":null,\"ipsec_static_calculate_delay_de_pci\":\"0000:0b:00.0\",\"gw_ver\":\"ipv4\",\"ike_ver\":\"ikev2\",\"kmp_dh_group\":\"modp1024\",\"kmp_hash_alg\":\"hmac_md5\",\"kmp_enc_alg\":\"3des_cbc\",\"sm4_type\":null,\"kmp_auth_method\":\"psk\",\"kmp_auth_rsa_padding\":\"pkcs1\",\"isakmp_id_type\":\"fqdn\",\"srv_isakmp_id_type\":\"fqdn\",\"kmp_prf_alg\":\"hmac_sha1\",\"tunnel\":\"1\",\"ipsec_flow_start_interval\":\"0\",\"ipsec_dpd_interval\":\"0\",\"ipsec_nonce_size\":\"32\",\"ipsec_frag_size\":\"0\",\"send_timeout\":\"5\",\"max_retransmit\":\"5\",\"kmp_sa_nego_time_limit\":\"60\",\"ike_lifetime\":\"28800\",\"default_sport\":\"1025\",\"ipsec_transform\":\"esp\",\"esp_auth_alg\":\"hmac_sha1\",\"esp_enc_alg\":\"3des_cbc\",\"anti_replay_window\":\"0\",\"ah_auth_alg\":\"hmac_sha1\",\"sald_lifetime\":\"28800\",\"nat_keepalive_interval\":\"300\",\"nat_source_port\":\"4500\",\"nat_dest_port\":\"4500\",\"nat_orig_dest_port\":\"500\",\"branch_access_number\":\"1\",\"ike_taskgroup_interval\":\"1\",\"ipsec_modify_ftp_ip\":null,\"ipsec_modify_ftp_user\":null,\"ipsec_modify_ftp_password\":null,\"ipsec_modify_ftp_path\":null,\"traffic_select_type\":\"1\",\"resp_sip\":[null],\"resp_netmask\":[\"24\"],\"initiator_sip\":[null],\"initiator_netmask\":[\"24\"],\"traffic_select_file\":null,\"request_ip_enable\":\"1\",\"request_internal_addr\":\"1\",\"local_cert\":null,\"local_private\":null,\"local_enc_cert\":null,\"local_enc_private\":null,\"dest_cert\":null,\"dest_private\":null,\"dest_enc_cert\":null,\"dest_enc_private\":null,\"local_ca_cert\":null,\"dest_ca_cert\":null,\"xauth_username\":null,\"xauth_password\":null,\"eap_method_type\":\"eap_md5\",\"eap_identity_type\":\"fqdn\",\"eap_identity\":null,\"eap_password\":null,\"eap_vendor\":\"0\",\"eap_dest_cert\":null,\"eap_dest_private\":null,\"eap_local_ca_cert\":null,\"eap_dest_ca_cert\":null,\"tunnel_vlan_id\":null,\"tunnel_vlan_pri\":null,\"srv6_qinq_outer_vlan_id\":null,\"srv6_qinq_outer_vlan_pri\":null,\"srv6_qinq_inner_vlan_id\":null,\"srv6_qinq_inner_vlan_pri\":null,\"vxlan_vni\":null,\"vxlan_local_ip\":null,\"vxlan_remote_ip\":null,\"vxlan_local_port\":null,\"vxlan_remote_port\":null,\"vxlan_smac\":null,\"vxlan_dmac\":null,\"vxlan_qinq_outer_vlan_id\":null,\"vxlan_qinq_outer_vlan_pri\":null,\"vxlan_qinq_inner_vlan_id\":null,\"vxlan_qinq_inner_vlan_pri\":null,\"gre_ip\":null,\"gre_netmask\":null,\"gre_local_ip\":null,\"gre_src_ip\":null,\"gre_keyword_verify\":null,\"gre_mtu\":null,\"gre_keepalive_cycle\":null,\"gre_keepalive_num\":null,\"udproute_local_ip\":null,\"udproute_remote_ip\":null,\"udproute_local_port\":null,\"udproute_remote_port\":null,\"dns_type\":\"1\",\"dns_host_file\":null,\"dns_server_ip\":null,\"dns_standby_server_ip\":null,\"ipv4_tos_type\":\"0\",\"ipv4_tos\":\"0\",\"ipv4_tos_interval\":null,\"ipv4_tos2\":\"0\",\"ipv4_tos_time\":null,\"ipv4_tos2_time\":null,\"mac_config_enable\":\"0\",\"mac_config_prefix\":null,\"peer_mac_config_prefix\":null,\"enabled_vlan_ip\":\"0\",\"ipsec_static\":\"0\",\"ipsec_static_local_gw_hop\":\"0\",\"ipsec_static_remote_gw_hop\":\"0\",\"ipsec_static_sip_hop\":\"0\",\"ipsec_static_dip_hop\":\"0\",\"ipsec_static_in_esp_spi_random\":\"0\",\"ipsec_static_in_esp_iv_random\":\"0\",\"ipsec_static_in_enc_str_random\":\"0\",\"ipsec_static_in_auth_str_random\":\"0\",\"ipsec_static_in_padding_str_random\":\"0\",\"ipsec_static_out_esp_spi_random\":\"0\",\"ipsec_static_out_esp_iv_random\":\"0\",\"ipsec_static_out_enc_str_random\":\"0\",\"ipsec_static_out_auth_str_random\":\"0\",\"ipsec_static_out_padding_str_random\":\"0\",\"ipsec_static_calculate_delay_enable\":\"0\",\"remote_access\":\"0\",\"ipsec_verify_id\":\"0\",\"ipsec_frag_enabled\":\"0\",\"ext_sequence\":\"0\",\"pfs_dh_group\":\"0\",\"nat_traversal\":\"0\",\"branch_access\":\"0\",\"branch_access_remote_gw_jump\":\"0\",\"branch_access_src_jump\":\"0\",\"branch_access_dest_jump\":\"0\",\"ipsec_modify_enabled\":\"0\",\"ipsec_modify_python_log_enable\":\"0\",\"ipsec_modify_async_enable\":\"0\",\"ipsec_modify_recaculate_length\":\"0\",\"ipsec_modify_padding_enable\":\"0\",\"ipsec_modify_spi\":\"0\",\"enabled_tunnel_vlan\":\"0\",\"tunnel_srv6_enable\":\"0\",\"srv6_qinq_vlan_enable\":\"0\",\"vxlan_enable\":\"0\",\"vxlan_frag_enable\":\"0\",\"vxlan_qinq_vlan_enable\":\"0\",\"enabled_gre_vpn\":\"0\",\"gre_tunnel_verify\":\"0\",\"enabled_gre_keepalive\":\"0\",\"enabled_udproute\":\"0\",\"pppoe_session\":\"0\",\"eth_mpls_srv\":\"0\",\"srv_isakmp_id\":\"ISAKMPID\"}}",

Suggested Solutions

No response