Open Lirikana opened 5 months ago
Try disabling CUDA graphs --enforce-eager
, should help.
The same error occurs. It appears that one of the ray workers is trying to allocate way too much memory.
torch.cuda.OutOfMemoryError: CUDA out of memory.
Tried to allocate 512.00 MiB. GPU 0 has a total capacity of 22.00 GiB of which 0 bytes is free.
Process 232346 has 17179869184.00 GiB memory in use. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use.
Of the allocated memory 20.96 GiB is allocated by PyTorch, and 137.73 MiB is reserved by PyTorch but unallocated.
If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
You may need to lower your context length by specifying --max-model-len 4096
Different error after lowering context length.
*** SIGSEGV received at time=1711454288 on cpu 22 ***
@ 0x7f9340ccc520 1479940200 (unknown)
@ 0x2000 (unknown) (unknown)
@ 0x2 1528629200 (unknown)
@ 0x7f933f8d2e30 (unknown) (unknown)
@ 0xec8348fb89485355 (unknown) (unknown)
[2024-03-26 19:58:08,630 E 235769 235769] logging.cc:361: *** SIGSEGV received at time=1711454288 on cpu 22 ***
[2024-03-26 19:58:08,632 E 235769 235769] logging.cc:361: @ 0x7f9340ccc520 1479940200 (unknown)
[2024-03-26 19:58:08,634 E 235769 235769] logging.cc:361: @ 0x2000 (unknown) (unknown)
[2024-03-26 19:58:08,637 E 235769 235769] logging.cc:361: @ 0x2 1528629200 (unknown)
[2024-03-26 19:58:08,638 E 235769 235769] logging.cc:361: @ 0x7f933f8d2e30 (unknown) (unknown)
[2024-03-26 19:58:08,640 E 235769 235769] logging.cc:361: @ 0xec8348fb89485355 (unknown) (unknown)
Fatal Python error: Segmentation fault
Stack (most recent call first):
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/modeling/layers/quantization/gguf.py", line 135 in apply_weights
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/modeling/layers/vocab_parallel_embedding.py", line 170 in forward
File "/home/lirikana/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520 in _call_impl
File "/home/lirikana/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511 in _wrapped_call_impl
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/modeling/models/llama.py", line 422 in sample
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/task_handler/model_runner.py", line 700 in execute_model
File "/home/lirikana/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115 in decorate_context
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/task_handler/model_runner.py", line 758 in profile_run
File "/home/lirikana/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115 in decorate_context
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/task_handler/worker.py", line 136 in profile_num_available_blocks
File "/home/lirikana/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115 in decorate_context
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py", line 1028 in _run_workers
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py", line 321 in _init_cache
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py", line 118 in __init__
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/engine/async_aphrodite.py", line 410 in _init_engine
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/engine/async_aphrodite.py", line 341 in __init__
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/engine/async_aphrodite.py", line 676 in from_engine_args
File "/home/lirikana/.local/lib/python3.10/site-packages/aphrodite/endpoints/openai/api_server.py", line 563 in <module>
File "/usr/lib/python3.10/runpy.py", line 86 in _run_code
File "/usr/lib/python3.10/runpy.py", line 196 in _run_module_as_main
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, charset_normalizer.md, yaml._yaml, sentencepiece._sentencepiece, psutil._psutil_linux, psutil._psutil_posix, msgpack._cmsgpack, google._upb._message, setproctitle, ray._raylet, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy_backends.cuda.stream, cupy_backends.cuda.libs.cublas, cupy_backends.cuda.libs.cusolver, cupy_backends.cuda._softlink, cupy_backends.cuda.libs.cusparse, cupy._util, cupy.cuda.device, fastrlock.rlock, cupy.cuda.memory_hook, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.jitify, cupy.cuda.pinned_memory, cupy_backends.cuda.libs.curand, cupy_backends.cuda.libs.profiler, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._dtype, cupy._core._scalar, cupy._core._accelerator, cupy._core._memory_range, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupyx.cusolver, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._flinalg, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, cupy.cuda.cufft, cupy.fft._cache, cupy.fft._callback, cupy.random._generator_api, cupy.random._bit_generator, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, cupy.lib._polynomial, cupy_backends.cuda.libs.nccl, regex._regex, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, markupsafe._speedups, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct (total: 157)
Segmentation fault
I'm getting CUDA oom, the system has 128GB of system memory available.
Error appears to be happening on line 563 in api_server.py
engine = AsyncAphrodite.from_engine_args(engine_args)
Not sure where exactly it fails since efforts at putting in debug code in the format of
logger.debug()
doesn't yield any messages in the console.
Sorry I've been away for a while. Have you tried the docker image? This is probably a WSL issue. GPU docker on windows uses WSL too, but who knows...
Not sure how to run a docker image on windows. Hopefully official support for windows will be added in the future.
Your current environment
Please note that the system is WSL on windows 11, therefore the env.py is not able to gather the correct version of CUDA due to a known bug. The CUDA version installed is 12.1
🐛 Describe the bug
Trying to run a gguf model that has been converted to safetensors results in a Cuda out of memory error. This occurs after some of the ray workers have finished loading the model. 4 RTX2080ti 22GB were used which should have 88GB of VRAM available.
Launch Parameters
Error Log