BUG replica=2时，系统报错

官方最新docker镜像部署 xinference 0.10.3 当replica=2，GPU id 设置为2、3 时报错，详细情况如下：

024-05-31 03:04:47,135 xinference.core.worker 94 INFO You specify to launch the model: custom-chatglm3-6b-128k on GPU index: [2, 3] of the worker: 0.0.0.0:52727, xinference will automatically ignore the n_gpu option. 2024-05-31 03:04:52,032 xinference.model.llm.llm_family 94 INFO Caching from URI: /root/xcp/data/chatglm3-6b-128k 2024-05-31 03:04:52,033 xinference.model.llm.core 94 DEBUG Launching custom-chatglm3-6b-128k-2-0 with VLLMChatModel 2024-05-31 03:04:52,041 xinference.model.llm.vllm.core 111 INFO Loading custom-chatglm3-6b-128k with following model config: {'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 2, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None, 'max_model_len': 4096} 2024-05-31 03:04:54,075 WARNING services.py:1996 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67072000 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=10.24gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM. 2024-05-31 03:04:55,156 INFO worker.py:1749 -- Started a local Ray instance. INFO 05-31 03:04:56 llm_engine.py:74] Initializing an LLM engine (v0.4.0.post1) with config: model='/root/.xinference/cache/custom-chatglm3-6b-128k-pytorch-6b', tokenizer='/root/.xinference/cache/custom-chatglm3-6b-128k-pytorch-6b', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, seed=0) WARNING 05-31 03:04:56 tokenizer.py:104] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. INFO 05-31 03:05:02 selector.py:51] Cannot use FlashAttention because the package is not found. Please install it for better performance. INFO 05-31 03:05:02 selector.py:25] Using XFormers backend. (RayWorkerVllm pid=5579) INFO 05-31 03:05:04 selector.py:51] Cannot use FlashAttention because the package is not found. Please install it for better performance. (RayWorkerVllm pid=5579) INFO 05-31 03:05:04 selector.py:25] Using XFormers backend. INFO 05-31 03:05:04 pynccl_utils.py:45] vLLM is using nccl==2.17.1 SIGBUS received at time=1717124706 on cpu 89 PC: @ 0x7f30ee60bb51 (unknown) (unknown) (RayWorkerVllm pid=5579) SIGBUS received at time=1717124706 on cpu 24 (RayWorkerVllm pid=5579) PC: @ 0x7f026cd47b51 (unknown) (unknown) (RayWorkerVllm pid=5579) @ 0x7f026cf1c420 977573712 (unknown) (RayWorkerVllm pid=5579) @ 0x70752d6c63636e2f (unknown) (unknown) (RayWorkerVllm pid=5579) [2024-05-31 03:05:06,827 E 5579 5579] logging.cc:365: SIGBUS received at time=1717124706 on cpu 24 (RayWorkerVllm pid=5579) [2024-05-31 03:05:06,828 E 5579 5579] logging.cc:365: PC: @ 0x7f026cd47b51 (unknown) (unknown) (RayWorkerVllm pid=5579) [2024-05-31 03:05:06,828 E 5579 5579] logging.cc:365: @ 0x7f026cf1c420 977573712 (unknown) (RayWorkerVllm pid=5579) [2024-05-31 03:05:06,830 E 5579 5579] logging.cc:365: @ 0x70752d6c63636e2f (unknown) (unknown) (RayWorkerVllm pid=5579) Fatal Python error: Bus error (RayWorkerVllm pid=5579) (RayWorkerVllm pid=5579) Stack (most recent call first): (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/pynccl.py", line 241 in init (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/pynccl_utils.py", line 46 in init_process_group (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 287 in init_distributed_environment (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 100 in init_device (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/vllm/engine/ray_utils.py", line 37 in execute_method (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/ray/util/tracing/tracing_helper.py", line 467 in _resume_span (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/ray/_private/function_manager.py", line 691 in actor_method_executor (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/ray/_private/worker.py", line 876 in main_loop (RayWorkerVllm pid=5579) File "/opt/conda/lib/python3.10/site-packages/ray/_private/workers/default_worker.py", line 289 in (RayWorkerVllm pid=5579) (RayWorkerVllm pid=5579) Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, _brotli, simplejson._speedups, uvloop.loop, ray._raylet, mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, sentencepiece._sentencepiece, pyarrow.lib, pyarrow._json, PIL._imaging, zstandard.backend_c, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._direct (total: 97) @ 0x7f30ee7e0420 (unknown) (unknown) @ 0x7a302d6c63636e2f (unknown) (unknown) [2024-05-31 03:05:07,155 E 111 111] logging.cc:365: SIGBUS received at time=1717124707 on cpu 89 [2024-05-31 03:05:07,160 E 111 111] logging.cc:365: PC: @ 0x7f30ee60bb51 (unknown) (unknown) [2024-05-31 03:05:07,160 E 111 111] logging.cc:365: @ 0x7f30ee7e0420 (unknown) (unknown) [2024-05-31 03:05:07,164 E 111 111] logging.cc:365: @ 0x7a302d6c63636e2f (unknown) (unknown) Fatal Python error: Bus error

Stack (most recent call first): File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/pynccl.py", line 241 in init File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/parallel_utils/pynccl_utils.py", line 46 in init_process_group File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 287 in init_distributed_environment File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 100 in init_device File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 324 in _run_workers File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 191 in _init_workers_ray File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 62 in init File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 110 in init File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 422 in _init_engine File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 311 in init File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 348 in from_engine_args File "/opt/conda/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 178 in load File "/opt/conda/lib/python3.10/site-packages/xinference/core/model.py", line 239 in load File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384 in __on_receive__ File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370 in _run_coro File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659 in send File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370 in _run_coro File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 422 in process_message File "/opt/conda/lib/python3.10/asyncio/events.py", line 80 in _run File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909 in _run_once File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603 in run_forever File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636 in run_until_complete File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44 in run File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/indigen/pool.py", line 278 in _start_sub_pool File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108 in run File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129 in _main File "/opt/conda/lib/python3.10/multiprocessing/forkserver.py", line 313 in _serve_one File "/opt/conda/lib/python3.10/multiprocessing/forkserver.py", line 274 in main File "", line 1 in

Extension modules: xoscar.context, xoscar.core, xoscar._utils, mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, xoscar.serialization.core, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, xoscar.backends.message, psutil._psutil_linux, psutil._psutil_posix, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, markupsafe._speedups, _brotli, zstandard.backend_c, simplejson._speedups, yaml._yaml, PIL._imaging, sentencepiece._sentencepiece, PIL._imagingft, regex._regex, msgpack._cmsgpack, google._upb._message, setproctitle, uvloop.loop, ray._raylet, pyarrow._json, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._direct (total: 146) 2024-05-31 03:05:12,371 xinference.core.worker 94 ERROR Failed to load model custom-chatglm3-6b-128k-2-0 Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 697, in launch_builtin_model await model_ref.load() File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 226, in send result = await self._wait(future, actor_ref.address, send_message) # type: ignore File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 115, in _wait return await future File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/core.py", line 84, in _listen raise ServerClosed( xoscar.errors.ServerClosed: Remote server unixsocket:///12320768 closed 2024-05-31 03:05:12,412 xinference.core.supervisor 94 DEBUG Enter terminate_model, args: (<xinference.core.supervisor.SupervisorActor object at 0x7fad664fe610>, 'custom-chatglm3-6b-128k'), kwargs: {'suppress_exception': True} 2024-05-31 03:05:12,413 xinference.core.supervisor 94 DEBUG Leave terminate_model, elapsed time: 0 s 2024-05-31 03:05:12,416 xinference.api.restful_api 1 ERROR [address=0.0.0.0:52727, pid=94] Remote server unixsocket:///12320768 closed Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/xinference/api/restful_api.py", line 731, in launch_model model_uid = await (await self._get_supervisor_ref()).launch_builtin_model( File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send result = await self._run_coro(message.message_id, coro) File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro return await coro File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive return await super().on_receive(message) # type: ignore File "xoscar/core.pyx", line 558, in on_receive__ raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive async with self._lock: File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive with debug_async_timeout('actor_lock_timeout', File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.__on_receive result = await result File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 850, in launch_builtin_model await _launch_model() File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 814, in _launch_model await _launch_one_model(rep_model_uid) File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 796, in _launch_one_model await worker_ref.launch_builtin_model( File "xoscar/core.pyx", line 284, in __pyx_actor_method_wrapper async with lock: File "xoscar/core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper result = await result File "/opt/conda/lib/python3.10/site-packages/xinference/core/utils.py", line 45, in wrapped ret = await func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 697, in launch_builtin_model await model_ref.load() File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 226, in send result = await self._wait(future, actor_ref.address, send_message) # type: ignore File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 115, in _wait return await future File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/core.py", line 84, in _listen raise ServerClosed( xoscar.errors.ServerClosed: [address=0.0.0.0:52727, pid=94] Remote server unixsocket:///12320768 closed

xorbitsai / inference

BUG replica=2时，系统报错 #1580