Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
qinxuye
commented
2 hours ago
System Info / 系統信息
Name Version Build Channel
accelerate 1.0.1 pypi_0 pypi aiofiles 23.2.1 pypi_0 pypi aiohappyeyeballs 2.4.3 pypi_0 pypi aiohttp 3.10.10 pypi_0 pypi aioprometheus 23.12.0 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi airportsdata 20241001 pypi_0 pypi aliyun-python-sdk-core 2.16.0 pypi_0 pypi aliyun-python-sdk-kms 2.16.5 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi antlr4-python3-runtime 4.9.3 pypi_0 pypi anyio 4.6.2.post1 pypi_0 pypi aom 3.9.1 he0c23c2_0 conda-forge argcomplete 3.5.1 pypi_0 pypi async-timeout 4.0.3 pypi_0 pypi attrdict 2.0.1 pypi_0 pypi attrs 24.2.0 pypi_0 pypi audioread 3.0.1 pypi_0 pypi auto-gptq 0.7.1 pypi_0 pypi autoawq 0.2.5 pypi_0 pypi autoawq-kernels 0.0.8 pypi_0 pypi av 13.1.0 pypi_0 pypi bcrypt 4.2.0 pypi_0 pypi beautifulsoup4 4.12.3 pypi_0 pypi bitsandbytes 0.44.1 pypi_0 pypi black 24.10.0 pypi_0 pypi blas 1.0 mkl boto3 1.28.64 pypi_0 pypi botocore 1.31.85 pypi_0 pypi brotli-python 1.0.9 py311hd77b12b_8 bzip2 1.0.8 h2bbff1b_6 ca-certificates 2024.9.24 haa95532_0 cairo 1.18.0 h32b962e_3 conda-forge cdifflib 1.2.6 pypi_0 pypi certifi 2024.8.30 pyhd8ed1ab_0 conda-forge cffi 1.17.1 pypi_0 pypi charset-normalizer 3.3.2 pyhd3eb1b0_0 chattts 0.2.0 pypi_0 pypi click 8.1.7 pypi_0 pypi cloudpickle 3.1.0 pypi_0 pypi colorama 0.4.6 pypi_0 pypi coloredlogs 15.0.1 pypi_0 pypi conformer 0.3.2 pypi_0 pypi contourpy 1.3.0 pypi_0 pypi controlnet-aux 0.0.7 pypi_0 pypi crcmod 1.7 pypi_0 pypi cryptography 43.0.3 pypi_0 pypi cuda-cccl 12.6.77 0 nvidia cuda-cccl_win-64 12.6.77 0 nvidia cuda-cudart 12.1.105 0 nvidia cuda-cudart-dev 12.1.105 0 nvidia cuda-cupti 12.1.105 0 nvidia cuda-libraries 12.1.0 0 nvidia cuda-libraries-dev 12.1.0 0 nvidia cuda-nvrtc 12.1.105 0 nvidia cuda-nvrtc-dev 12.1.105 0 nvidia cuda-nvtx 12.1.105 0 nvidia cuda-opencl 12.6.77 0 nvidia cuda-opencl-dev 12.6.77 0 nvidia cuda-profiler-api 12.6.77 0 nvidia cuda-runtime 12.1.0 0 nvidia cuda-version 12.6 3 nvidia cycler 0.12.1 pypi_0 pypi cython 3.0.11 pypi_0 pypi datamodel-code-generator 0.26.2 pypi_0 pypi datasets 2.14.4 pypi_0 pypi dav1d 1.2.1 hcfcfb64_0 conda-forge decorator 5.1.1 pypi_0 pypi diffusers 0.31.0 pypi_0 pypi dill 0.3.7 pypi_0 pypi diskcache 5.6.3 pypi_0 pypi distro 1.9.0 pypi_0 pypi dlfcn-win32 1.4.1 h63175ca_0 conda-forge dnspython 2.7.0 pypi_0 pypi ecdsa 0.19.0 pypi_0 pypi editdistance 0.8.1 pypi_0 pypi einops 0.8.0 pypi_0 pypi einx 0.3.0 pypi_0 pypi email-validator 2.2.0 pypi_0 pypi encodec 0.1.1 pypi_0 pypi eva-decord 0.6.1 pypi_0 pypi expat 2.6.3 he0c23c2_0 conda-forge fastapi 0.115.3 pypi_0 pypi ffmpeg 6.1.2 gpl_hab0cf1a_505 conda-forge ffmpy 0.4.0 pypi_0 pypi filelock 3.13.1 py311haa95532_0 flagembedding 1.2.11 pypi_0 pypi flatbuffers 24.3.25 pypi_0 pypi font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_3 conda-forge fontconfig 2.14.2 hbde0cde_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.54.1 pypi_0 pypi freetype 2.12.1 hdaf720e_2 conda-forge fribidi 1.0.10 h8d14728_0 conda-forge frozendict 2.4.6 pypi_0 pypi frozenlist 1.5.0 pypi_0 pypi fsspec 2024.10.0 pypi_0 pypi funasr 1.1.12 pypi_0 pypi fvcore 0.1.5.post20221221 pypi_0 pypi gdown 5.2.0 pypi_0 pypi gekko 1.2.1 pypi_0 pypi genson 1.3.0 pypi_0 pypi getopt-win32 0.1 hcfcfb64_1 conda-forge giflib 5.2.2 h7edc060_0 glib 2.82.2 h7025463_0 conda-forge glib-tools 2.82.2 h4394cf3_0 conda-forge gmpy2 2.1.2 py311h7f96b67_0 gradio 5.4.0 pypi_0 pypi gradio-client 1.4.2 pypi_0 pypi graphite2 1.3.13 h63175ca_1003 conda-forge graphviz 7.1.0 h51cb2cd_0 conda-forge gts 0.7.6 h6b5321d_4 conda-forge h11 0.14.0 pypi_0 pypi harfbuzz 9.0.0 h2bedf89_1 conda-forge hiredis 3.0.0 pypi_0 pypi httpcore 1.0.6 pypi_0 pypi httpx 0.27.2 pypi_0 pypi huggingface-hub 0.26.1 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi hydra-core 1.3.2 pypi_0 pypi hyperpyyaml 1.2.2 pypi_0 pypi icu 75.1 he0c23c2_0 conda-forge idna 3.7 py311haa95532_0 imageio 2.36.0 pypi_0 pypi imageio-ffmpeg 0.5.1 pypi_0 pypi importlib-metadata 8.5.0 pypi_0 pypi importlib-resources 6.4.5 pypi_0 pypi inflect 5.6.2 pypi_0 pypi intel-openmp 2023.1.0 h59b6b97_46320 interegular 0.3.3 pypi_0 pypi iopath 0.1.10 pypi_0 pypi isort 5.13.2 pypi_0 pypi jaconv 0.4.0 pypi_0 pypi jamo 0.4.1 pypi_0 pypi jieba 0.42.1 pypi_0 pypi jinja2 3.1.4 py311haa95532_0 jiter 0.6.1 pypi_0 pypi jj-pytorchvideo 0.1.5 pypi_0 pypi jmespath 0.10.0 pypi_0 pypi joblib 1.4.2 pypi_0 pypi jpeg 9e h827c3e9_3 jsonschema 4.23.0 pypi_0 pypi jsonschema-specifications 2024.10.1 pypi_0 pypi kaldiio 2.18.0 pypi_0 pypi kiwisolver 1.4.7 pypi_0 pypi lark 1.2.2 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi lcms2 2.12 h83e58a3_0 lerc 3.0 hd77b12b_0 libcublas 12.1.0.26 0 nvidia libcublas-dev 12.1.0.26 0 nvidia libcufft 11.0.2.4 0 nvidia libcufft-dev 11.0.2.4 0 nvidia libcurand 10.3.7.77 0 nvidia libcurand-dev 10.3.7.77 0 nvidia libcusolver 11.4.4.55 0 nvidia libcusolver-dev 11.4.4.55 0 nvidia libcusparse 12.0.2.55 0 nvidia libcusparse-dev 12.0.2.55 0 nvidia libdeflate 1.17 h2bbff1b_1 libexpat 2.6.3 he0c23c2_0 conda-forge libffi 3.4.4 hd77b12b_1 libgd 2.3.3 ha43c60c_1 libglib 2.82.2 h7025463_0 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge libintl 0.22.5 h5728263_3 conda-forge libintl-devel 0.22.5 h5728263_3 conda-forge libjpeg-turbo 2.0.0 h196d8e1_0 libnacl 2.1.0 pypi_0 pypi libnpp 12.0.2.50 0 nvidia libnpp-dev 12.0.2.50 0 nvidia libnvjitlink 12.1.105 0 nvidia libnvjitlink-dev 12.1.105 0 nvidia libnvjpeg 12.1.1.14 0 nvidia libnvjpeg-dev 12.1.1.14 0 nvidia libopus 1.3.1 h8ffe710_1 conda-forge libpng 1.6.44 h3ca93ac_0 conda-forge librosa 0.10.2.post1 pypi_0 pypi libsqlite 3.47.0 h2466b09_0 conda-forge libtiff 4.4.0 h8a3f274_2 libuv 1.48.0 h827c3e9_0 libwebp 1.3.2 hbc33d0d_0 libwebp-base 1.3.2 h3d04722_1 libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.12.7 h0f24e4e_4 conda-forge libzlib 1.3.1 h2466b09_2 conda-forge lightning 2.4.0 pypi_0 pypi lightning-utilities 0.11.8 pypi_0 pypi llama-cpp-python 0.3.1 pypi_0 pypi llvmlite 0.43.0 pypi_0 pypi loguru 0.7.2 pypi_0 pypi loralib 0.1.2 pypi_0 pypi lz4-c 1.9.4 h2bbff1b_1 m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown-it-py 3.0.0 pypi_0 pypi markupsafe 2.1.3 py311h2bbff1b_0 matplotlib 3.9.2 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mkl 2023.1.0 h6b88ed4_46358 mkl-service 2.4.0 py311h2bbff1b_1 mkl_fft 1.3.10 py311h827c3e9_0 mkl_random 1.2.7 py311hea22821_0 modelscope 1.19.2 pypi_0 pypi mpc 1.1.0 h7edee0f_1 mpfr 4.0.2 h62dcd97_1 mpir 3.0.0 hec2e145_1 mpmath 1.3.0 py311haa95532_0 msgpack 1.1.0 pypi_0 pypi msys2-conda-epoch 20160418 1 conda-forge multidict 6.1.0 pypi_0 pypi multiprocess 0.70.15 pypi_0 pypi mypy-extensions 1.0.0 pypi_0 pypi natsort 8.4.0 pypi_0 pypi nemo-text-processing 1.0.2 pypi_0 pypi nest-asyncio 1.6.0 pypi_0 pypi networkx 3.2.1 py311haa95532_0 numba 0.60.0 pypi_0 pypi numpy 1.26.4 pypi_0 pypi nvidia-ml-py 12.560.30 pypi_0 pypi omegaconf 2.3.0 pypi_0 pypi onnxruntime 1.16.0 pypi_0 pypi openai 1.52.2 pypi_0 pypi opencv-python 4.10.0.84 pypi_0 pypi openfst 1.8.2 h91493d7_2 conda-forge openh264 2.4.1 h63175ca_0 conda-forge openjpeg 2.5.0 hc9384bd_1 conda-forge openssl 3.3.2 h2466b09_0 conda-forge optimum 1.23.2 pypi_0 pypi orjson 3.10.10 pypi_0 pypi ormsgpack 1.6.0 pypi_0 pypi oss2 2.19.1 pypi_0 pypi outlines 0.1.1 pypi_0 pypi outlines-core 0.1.14 pypi_0 pypi packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.2.3 pypi_0 pypi pango 1.54.0 hbb871f6_2 conda-forge parameterized 0.9.0 pypi_0 pypi passlib 1.7.4 pypi_0 pypi pathspec 0.12.1 pypi_0 pypi pcre2 10.44 h3d7b363_2 conda-forge peft 0.13.2 pypi_0 pypi pillow 11.0.0 pypi_0 pypi pip 24.2 py311haa95532_0 pixman 0.43.4 h63175ca_0 conda-forge platformdirs 4.3.6 pypi_0 pypi pooch 1.8.2 pypi_0 pypi portalocker 2.10.1 pypi_0 pypi propcache 0.2.0 pypi_0 pypi protobuf 5.28.3 pypi_0 pypi psutil 6.1.0 pypi_0 pypi pthread-stubs 0.4 hcd874cb_1001 conda-forge pyarrow 17.0.0 pypi_0 pypi pyasn1 0.6.1 pypi_0 pypi pybase16384 0.3.7 pypi_0 pypi pycountry 24.6.1 pypi_0 pypi pycparser 2.22 pypi_0 pypi pycryptodome 3.21.0 pypi_0 pypi pydantic 2.9.2 pypi_0 pypi pydantic-core 2.23.4 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pygments 2.18.0 pypi_0 pypi pynini 2.1.5 py311h005e61a_6 conda-forge pynndescent 0.5.13 pypi_0 pypi pyparsing 3.2.0 pypi_0 pypi pyreadline3 3.5.4 pypi_0 pypi pysocks 1.7.1 py311haa95532_0 python 3.11.10 hce54a09_3_cpython conda-forge python-dateutil 2.9.0.post0 pypi_0 pypi python-jose 3.3.0 pypi_0 pypi python-multipart 0.0.12 pypi_0 pypi python_abi 3.11 2_cp311 conda-forge pytorch 2.5.0 py3.11_cuda12.1_cudnn9_0 pytorch pytorch-cuda 12.1 hde6ce7c_6 pytorch pytorch-lightning 2.4.0 pypi_0 pypi pytorch-mutex 1.0 cuda pytorch pytorch-wpe 0.0.1 pypi_0 pypi pytz 2024.2 pypi_0 pypi pywin32 308 pypi_0 pypi pyyaml 6.0.2 py311h827c3e9_0 quantile-python 1.1 pypi_0 pypi qwen-vl-utils 0.0.8 pypi_0 pypi redis 5.2.0 pypi_0 pypi referencing 0.35.1 pypi_0 pypi regex 2024.9.11 pypi_0 pypi requests 2.32.3 py311haa95532_0 rich 13.9.3 pypi_0 pypi rouge 1.0.1 pypi_0 pypi rpds-py 0.20.0 pypi_0 pypi rsa 4.9 pypi_0 pypi ruamel-yaml 0.18.6 pypi_0 pypi ruamel-yaml-clib 0.2.12 pypi_0 pypi ruff 0.7.1 pypi_0 pypi s3transfer 0.7.0 pypi_0 pypi sacremoses 0.1.1 pypi_0 pypi safehttpx 0.1.1 pypi_0 pypi safetensors 0.4.5 pypi_0 pypi scikit-image 0.24.0 pypi_0 pypi scikit-learn 1.5.2 pypi_0 pypi scipy 1.14.1 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi sentence-transformers 3.2.1 pypi_0 pypi sentencepiece 0.2.0 pypi_0 pypi setuptools 75.1.0 py311haa95532_0 shellingham 1.5.4 pypi_0 pypi six 1.16.0 pypi_0 pypi sniffio 1.3.1 pypi_0 pypi soundfile 0.12.1 pypi_0 pypi soupsieve 2.6 pypi_0 pypi soxr 0.5.0.post1 pypi_0 pypi sqlite 3.45.3 h2bbff1b_0 sse-starlette 2.1.3 pypi_0 pypi starlette 0.41.0 pypi_0 pypi svt-av1 2.2.1 he0c23c2_0 conda-forge sympy 1.13.1 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tbb 2021.8.0 h59b6b97_0 tblib 3.0.0 pypi_0 pypi tensorboardx 2.6.2.2 pypi_0 pypi tensorizer 2.9.0 pypi_0 pypi termcolor 2.5.0 pypi_0 pypi threadpoolctl 3.5.0 pypi_0 pypi tifffile 2024.9.20 pypi_0 pypi tiktoken 0.8.0 pypi_0 pypi timm 1.0.11 pypi_0 pypi tk 8.6.13 h5226925_1 conda-forge tokenizers 0.20.1 pypi_0 pypi tomlkit 0.12.0 pypi_0 pypi torch-complex 0.4.4 pypi_0 pypi torchaudio 2.5.0 pypi_0 pypi torchmetrics 1.5.1 pypi_0 pypi torchvision 0.20.0 pypi_0 pypi tqdm 4.66.5 pypi_0 pypi transformers 4.46.0 pypi_0 pypi transformers-stream-generator 0.0.5 pypi_0 pypi typer 0.12.5 pypi_0 pypi typing_extensions 4.11.0 py311haa95532_0 tzdata 2024.2 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_1 conda-forge umap-learn 0.5.6 pypi_0 pypi urllib3 2.0.7 pypi_0 pypi uvicorn 0.32.0 pypi_0 pypi vc 14.40 h2eaa2aa_1 vc14_runtime 14.40.33810 hcc2c482_22 conda-forge vector-quantize-pytorch 1.18.5 pypi_0 pypi verovio 4.3.1 pypi_0 pypi vocos 0.1.0 pypi_0 pypi vs2015_runtime 14.40.33810 h3bf8584_22 conda-forge websockets 12.0 pypi_0 pypi wetextprocessing 1.0.3 pypi_0 pypi wget 3.2 pypi_0 pypi wheel 0.44.0 py311haa95532_0 win32-setctime 1.1.0 pypi_0 pypi win_inet_pton 1.1.0 py311haa95532_0 wrapt 1.16.0 pypi_0 pypi x264 1!164.3095 h8ffe710_2 conda-forge x265 3.5 h2d74725_3 conda-forge xinference 0.16.1 pypi_0 pypi xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xoscar 0.3.3 pypi_0 pypi xxhash 3.5.0 pypi_0 pypi xz 5.4.6 h8cc25b3_1 yacs 0.1.8 pypi_0 pypi yaml 0.2.5 he774522_0 yarl 1.16.0 pypi_0 pypi zipp 3.20.2 pypi_0 pypi zlib 1.3.1 h2466b09_2 conda-forge zstandard 0.23.0 pypi_0 pypi zstd 1.5.6 h0ea2cb4_0 conda-forge
Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？
Version info / 版本信息
xinference 0.16.1
The command used to start Xinference / 用以启动 xinference 的命令
xinference-local --host 127.0.0.1 --port 9997
Reproduction / 复现过程
conda create -n xinference python=3.11.7 conda activate xinference conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia conda install -c conda-forge pynini=2.1.5 conda install -c conda-forge ffmpeg=6 pip install "xinference[all]" pip install matplotlib pip install --upgrade Pillow xinference-local --host 127.0.0.1 --port 9997 xinference launch --model-name CosyVoice-300M-SFT --model-type audio --gpu-idx 0 xinference launch --model-name SenseVoiceSmall --model-type audio --gpu-idx 0
Expected behavior / 期待表现
dify中CosyVoice-300M-SFT 能够播放声音 dify中SenseVoiceSmall 不报错
[xinference] Server Unavailable Error, Failed to transcribe the audio, detail: [address=127.0.0.1:11483, pid=24084] Failed to load audio: ffmpeg version 6.1.2 Copyright (c) 2000-2024 the FFmpeg developers built with clang version 19.1.0 configuration: --prefix=/d/bld/ffmpeg_1726960534850/_h_env/Library --cc=clang.exe --cxx=clang++.exe --nm=llvm-nm --ar=llvm-ar --disable-doc --enable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libharfbuzz --enable-libfontconfig --enable-libopenh264 --enable-libdav1d --ld=lld-link --target-os=win64 --enable-cross-compile --toolchain=msvc --host-cc=clang.exe --extra-libs=ucrt.lib --extra-libs=vcruntime.lib --extra-libs=oldnames.lib --strip=llvm-strip --disable-stripping --host-extralibs= --disable-libopenvino --enable-gpl --enable-libx264 --enable-libx265 --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/d/bld/ffmpeg_1726960534850/_build_env/Library/bin/pkg-config libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100 [in#0 @ 0000027A1E33BE40] Error opening input: Permission denied Error opening input file C:\Users\85390\AppData\Local\Temp\tmprgx4on2u. Error opening input files: Permission denied