Closed sharlynxy closed 16 minutes ago
I had the same problem
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
[ubuntu-Precision-7960-Tower:16603] *** Process received signal ***
[ubuntu-Precision-7960-Tower:16603] Signal: Segmentation fault (11)
[ubuntu-Precision-7960-Tower:16603] Signal code: Address not mapped (1)
[ubuntu-Precision-7960-Tower:16603] Failing at address: 0x440000e9
[ubuntu-Precision-7960-Tower:16603] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7b630b242520]
[ubuntu-Precision-7960-Tower:16603] [ 1] /lib/x86_64-linux-gnu/libmpi.so.40(PMPI_Comm_set_errhandler+0x47)[0x7b61e87366b7]
[ubuntu-Precision-7960-Tower:16603] [ 2] /home/ubuntu/anaconda3/envs/trtllm/lib/python3.10/site-packages/mpi4py/MPI.cpython-310-x86_64-linux-gnu.so(+0x9b550)[0x7b60d2f3c550]
[ubuntu-Precision-7960-Tower:16603] [ 3] /home/ubuntu/anaconda3/envs/trtllm/lib/python3.10/site-packages/mpi4py/MPI.cpython-310-x86_64-linux-gnu.so(+0x2e41f)[0x7b60d2ecf41f]
[ubuntu-Precision-7960-Tower:16603] [ 4] python(PyModule_ExecDef+0x70)[0x597d40]
[ubuntu-Precision-7960-Tower:16603] [ 5] python[0x5990c9]
[ubuntu-Precision-7960-Tower:16603] [ 6] python[0x4fd37b]
[ubuntu-Precision-7960-Tower:16603] [ 7] python(_PyEval_EvalFrameDefault+0x5a74)[0x4f37a4]
[ubuntu-Precision-7960-Tower:16603] [ 8] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [ 9] python(_PyEval_EvalFrameDefault+0x4b26)[0x4f2856]
[ubuntu-Precision-7960-Tower:16603] [10] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [11] python(_PyEval_EvalFrameDefault+0x731)[0x4ee461]
[ubuntu-Precision-7960-Tower:16603] [12] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [13] python(_PyEval_EvalFrameDefault+0x31f)[0x4ee04f]
[ubuntu-Precision-7960-Tower:16603] [14] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [15] python(_PyEval_EvalFrameDefault+0x31f)[0x4ee04f]
[ubuntu-Precision-7960-Tower:16603] [16] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [17] python[0x4fd514]
[ubuntu-Precision-7960-Tower:16603] [18] python(_PyObject_CallMethodIdObjArgs+0x137)[0x50c327]
[ubuntu-Precision-7960-Tower:16603] [19] python(PyImport_ImportModuleLevelObject+0x525)[0x50b685]
[ubuntu-Precision-7960-Tower:16603] [20] python[0x517454]
[ubuntu-Precision-7960-Tower:16603] [21] python[0x4fd907]
[ubuntu-Precision-7960-Tower:16603] [22] python(PyObject_Call+0x209)[0x50a259]
[ubuntu-Precision-7960-Tower:16603] [23] python(_PyEval_EvalFrameDefault+0x5a74)[0x4f37a4]
[ubuntu-Precision-7960-Tower:16603] [24] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [25] python(_PyEval_EvalFrameDefault+0x31f)[0x4ee04f]
[ubuntu-Precision-7960-Tower:16603] [26] python(_PyFunction_Vectorcall+0x6f)[0x4fdd4f]
[ubuntu-Precision-7960-Tower:16603] [27] python[0x4fd514]
[ubuntu-Precision-7960-Tower:16603] [28] python(_PyObject_CallMethodIdObjArgs+0x137)[0x50c327]
[ubuntu-Precision-7960-Tower:16603] [29] python(PyImport_ImportModuleLevelObject+0x9da)[0x50bb3a]
[ubuntu-Precision-7960-Tower:16603] *** End of error message ***
[1] 16603 segmentation fault (core dumped) python app.py
And I add
import faulthandler
faulthandler.enable()
get the error
Current thread 0x000076ef939e8740 (most recent call first):
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1184 in exec_module
File "<frozen importlib._bootstrap>", line 688 in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027 in _find_and_load
File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1078 in _handle_fromlist
File "/home/ubuntu/anaconda3/envs/trtllm/lib/python3.10/site-packages/tensorrt_llm/_utils.py", line 216 in mpi_comm
File "/home/ubuntu/anaconda3/envs/trtllm/lib/python3.10/site-packages/tensorrt_llm/_utils.py", line 221 in mpi_rank
File "/home/ubuntu/TensorRt_LLM/trt-llm-rag-linux-master/trt_llama_api.py", line 106 in __init__
File "/home/ubuntu/TensorRt_LLM/trt-llm-rag-linux-master/app.py", line 104 in <module>
Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, gmpy2.gmpy2, cython.cimports.libc.math, scipy._lib._ccallback_c, numpy.linalg.lapack_lite, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._flinalg, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, regex._regex, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, sklearn.utils._isfinite, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils.sparsefuncs_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.utils._vector_sentinel, sklearn.feature_extraction._hashing_fast, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, _brotli, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.hashing, pyarrow.lib, pyarrow._hdfsio, pandas._libs.tslib, pandas._libs.ops, pyarrow._compute, pandas._libs.arrays, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, yaml._yaml, sentencepiece._sentencepiece, cuda._lib.utils, cuda._cuda.ccuda, cuda.ccuda, cuda.cuda, cuda._lib.ccudart.utils, cuda._lib.ccudart.ccudart, cuda.ccudart, cuda.cudart, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, faiss._swigfaiss_avx2, websockets.speedups, PIL._imaging, ujson, matplotlib._c_internal_utils, matplotlib._path, kiwisolver._cext, markupsafe._speedups, matplotlib._image, PIL._imagingmath, PIL._webp, mpi4py.MPI (total: 243)
[1] 16874 segmentation fault (core dumped) python app.py
System Info
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
I used
pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com
to install trt-llmbuild engine with model
llama--Llama-2-7b-chat-hf
. command is following:python build.py --model_dir $PATH_TO_LLAMA2_CHAT_HF --dtype float16 \ --remove_input_padding \ --use_gpt_attention_plugin float16 \ --enable_context_fmha \ --use_gemm_plugin float16 \ --max_batch_size 1 \ --max_input_len 60 \ --max_output_len 60 \ --output_dir $PATH_TO_ENGINE
Then I run the example script with
python3 ../run.py --max_output_len=20 \ --tokenizer_dir $PATH_TO_LLAMA2_CHAT_HF \ --engine_dir=$PATH_TO_ENGINE
, then I gotExpected behavior
expect to run
TensortRT-LLM/examples/run.py
successully.actual behavior
outputs of building engine:
outputs of running the engine
additional notes
When I run the code as below I got the same problem.