Segmentation fault

[JZArray]

Hello guys, nice work! But I have a problem while running the demo.

loaded weight from ./pretrained_models/hallo/net.pth Segmentation fault (core dumped)

How do I solve this issue?

Thanks in advance!

[xumingw]

Could you please provide more details about the error?

[JZArray]

Could you please provide more details about the error?

Some weights of the model checkpoint were not used when initializing UNet2DConditionModel: ['conv_norm_out.bias, conv_norm_out.weight, conv_out.bias, conv_out.weight'] INFO:hallo.models.unet_3d:loaded temporal unet's pretrained weights from pretrained_models/stable-diffusion-v1-5/unet ... The config attributes {'center_input_sample': False} were passed to UNet3DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file. Load motion module params from pretrained_models/motion_module/mm_sd_v15_v2.ckpt INFO:hallo.models.unet_3d:Loaded 453.20928M-parameter motion module loaded weight from ./pretrained_models/hallo/net.pth Fatal Python error: Segmentation fault

Thread 0x00007f0ec00ff640 (most recent call first): File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 324 in wait File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 600 in wait File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 1009 in _bootstrap_inner File "/xxxx/anaconda3/envs/hallo/lib/python3.10/threading.py", line 966 in _bootstrap

Current thread 0x00007f126a8fa740 (most recent call first): File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456 in _conv_forward File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460 in forward File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/hallo/models/resnet.py", line 63 in forward File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520 in _call_impl File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511 in _wrapped_call_impl File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/hallo/models/face_locator.py", line 108 in forward File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520 in _call_impl File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511 in _wrapped_call_impl File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/hallo/animate/face_animate.py", line 342 in call File "/xxxx/anaconda3/envs/hallo/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115 in decorate_context File "/xxxx/hallo-main/scripts/inference.py", line 327 in inference_process File "/xxxx/hallo-main/scripts/inference.py", line 379 in

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, yaml._yaml, PIL._imaging, google._upb._message, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, regex._regex, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.enum, av.error, av.utils, av.option, av.descriptor, av.container.pyio, av.dictionary, av.format, av.stream, av.container.streams, av.sidedata.motionvectors, av.sidedata.sidedata, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.audio.fifo, av.filter.pad, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.audio.resampler, av.bitstream, kiwisolver._cext, _cffi_backend, PIL._imagingft, skimage._shared.geometry, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, skimage.measure._ccomp, insightface.thirdparty.face3d.mesh.cython.mesh_core_cython, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, PIL._imagingmath, scipy.signal._sigtools, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.signal._sosfilt, scipy.signal._spectral, scipy.signal._peak_finding_utils, soxr.cysoxr, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, msgpack._cmsgpack (total: 236) Segmentation fault (core dumped)

[subazinga]

Which GPU do you use? And can you also list the version of all packages?

[JZArray]

I am using A100. absl-py 2.1.0 accelerate 0.28.0 albucore 0.0.11 albumentations 1.4.8 alembic 1.13.1 aniso8601 9.0.1 anna 0.1.0 annotated-types 0.7.0 antlr4-python3-runtime 4.9.3 astroid 3.2.2 attrs 23.2.0 audio-separator 0.17.2 audioread 3.0.1 av 12.1.0 beartype 0.18.5 bitsandbytes 0.43.1 blinker 1.8.2 bzip2 1.0.8 ca-certificates 2024.3.11 cachetools 5.3.3 certifi 2024.6.2 cffi 1.16.0 cfgv 3.4.0 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 coloredlogs 15.0.1 contextlib2 21.6.0 contourpy 1.2.1 cycler 0.12.1 Cython 3.0.10 decorator 4.4.2 decord 0.6.0 Deprecated 1.2.14 diffq 0.2.4 diffusers 0.27.2 dill 0.3.8 distlib 0.3.8 docker 7.1.0 easydict 1.13 einops 0.8.0 entrypoints 0.4 filelock 3.15.1 Flask 3.0.3 flatbuffers 24.3.25 fonttools 4.53.0 fsspec 2024.6.0 gitdb 4.0.11 GitPython 3.1.43 graphene 3.3 graphql-core 3.2.3 graphql-relay 3.2.0 greenlet 3.0.3 gunicorn 22.0.0 huggingface-hub 0.23.4
humanfriendly 10.0
identify 2.5.36
idna 3.7
imageio 2.34.1
imageio-ffmpeg 0.5.1
importlib_metadata 7.1.0
insightface 0.7.3
isort 5.13.2
itsdangerous 2.2.0
jax 0.4.29
jaxlib 0.4.29
Jinja2 3.1.4
joblib 1.4.2
julius 0.2.7
kiwisolver 1.4.5
lazy_loader 0.4
ld_impl_linux-64 2.38
libffi 3.3
libgcc-ng 9.1.0
librosa 0.10.2.post1
libstdcxx-ng 9.1.0
libuuid 1.0.3
llvmlite 0.43.0
Mako 1.3.5
Markdown 3.6
MarkupSafe 2.1.5
matplotlib 3.9.0
mccabe 0.7.0
mediapipe 0.10.14
ml-dtypes 0.4.0
ml_collections 0.1.1
mlflow 2.13.1
moviepy 1.0.3
mpmath 1.3.0
msgpack 1.0.8
ncurses 6.3
networkx 3.3
nodeenv 1.9.1
numba 0.60.0
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.19.3
nvidia-nvjitlink-cu12 12.5.40
nvidia-nvtx-cu12 12.1.105
omegaconf 2.3.0
onnx 1.16.1
onnx2torch 1.5.14
onnxruntime 1.18.0
opencv-contrib-python 4.9.0.80
opencv-python 4.9.0.80
opencv-python-headless 4.9.0.80
openssl 1.1.1w
opentelemetry-api 1.25.0
opentelemetry-sdk 1.25.0
opentelemetry-semantic-conventions 0.46b0
opt-einsum 3.3.0
packaging 24.1
pandas 2.2.2
pillow 10.3.0
pip 24.0
pip 24.0
platformdirs 4.2.2
pooch 1.8.2
pre-commit 3.7.1
prettytable 3.10.0
proglog 0.1.10
protobuf 4.25.3
psutil 5.9.8
pyarrow 15.0.2
pycparser 2.22
pydantic 2.7.4
pydantic_core 2.18.4
pydub 0.25.1
pylint 3.2.2
pyparsing 3.1.2
python 3.10.4
python-dateutil 2.9.0.post0
pytz 2024.1
PyYAML 6.0.1
querystring-parser 1.2.4
readline 8.1.2
regex 2024.5.15
requests 2.32.3
resampy 0.4.3
rotary-embedding-torch 0.6.2
safetensors 0.4.3
samplerate 0.1.0
scikit-image 0.23.2
scikit-learn 1.5.0
scipy 1.13.1
setuptools 69.5.1
setuptools 70.0.0
six 1.16.0
smmap 5.0.1
sounddevice 0.4.7
soundfile 0.12.1
soxr 0.3.7
SQLAlchemy 2.0.30
sqlite 3.38.5
sqlparse 0.5.0
sympy 1.12.1
threadpoolctl 3.5.0
tifffile 2024.5.22
tk 8.6.12
tokenizers 0.15.2
tomli 2.0.1
tomlkit 0.12.5
torch 2.2.2
torchvision 0.17.2
tqdm 4.66.4
transformers 4.39.2
triton 2.2.0
typing_extensions 4.12.2
tzdata 2024.1
tzdata 2024a
urllib3 2.2.1
virtualenv 20.26.2
wcwidth 0.2.13
Werkzeug 3.0.3
wheel 0.43.0
wheel 0.43.0
wrapt 1.16.0
xformers 0.0.25.post1
xz 5.2.5
zipp 3.19.2
zlib 1.2.12

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Tue_May__3_18:49:52_PDT_2022 Cuda compilation tools, release 11.7, V11.7.64 Build cuda_11.7.r11.7/compiler.31294372_0

CUDA Version: 12.2

@subazinga @xumingw

[JZArray]

solved

[gtfaiwxm]

how to solve the problem?

[JZArray]

how to solve the problem?

@gtfaiwxm I use torch-1.12, then it can work. I think it is due to the conflict between my CUDA(11.x) and torch 2.0.x in my case.

[gtfaiwxm]

Thanks.