segfault with pytorch 1.17 and 2.0.0

I have tried running IIM for awhile. So far no problem. First indication of problem after using pytorch 2.0.0, with CUDA 11.8

load pre_trained parameters for HR_Net
init weights done
  0%|                                                    | 0/48 [00:00<?, ?it/s]
 212img.jpg  image inference time:  3.727436065673828 133
  2%|▉                                           | 1/48 [00:03<02:55,  3.73s/it]Segmentation fault (core dumped)

No other error lines coming out. Any idea why?

Here is my full environment

Package                      Version              Editable project location
---------------------------- -------------------- --------------------------------------------------------------------------
absl-py                      1.4.0
adal                         1.2.7
addict                       2.4.0
aiofiles                     22.1.0
aiohttp                      3.8.4
aiokafka                     0.8.0
aiosignal                    1.3.1
albumentations               1.3.0
alembic                      1.10.4
antlr4-python3-runtime       4.9.3
anyio                        3.6.2
argcomplete                  2.1.2
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
arrow                        1.2.3
asgiref                      3.6.0
astor                        0.8.1
asttokens                    2.2.1
astunparse                   1.6.3
async-timeout                4.0.2
attrs                        22.2.0
av                           10.0.0
azure-common                 1.1.28
azure-core                   1.26.4
azure-graphrbac              0.61.1
azure-mgmt-authorization     3.0.0
azure-mgmt-containerregistry 10.1.0
azure-mgmt-core              1.4.0
azure-mgmt-keyvault          10.2.1
azure-mgmt-resource          22.0.0
azure-mgmt-storage           21.0.0
azureml-core                 1.50.0
Babel                        2.12.1
backcall                     0.2.0
backports.tempfile           1.0
backports.weakref            1.0.post1
basicsr                      1.3.4.4              /home/user/Desktop/CC_enhancements/PyDiff_LOL_4Sep23/PyDIff/BasicSR-light
bce-python-sdk               0.8.74
bcrypt                       4.0.1
beautifulsoup4               4.11.1
bleach                       5.0.1
blinker                      1.6.2
boto3                        1.26.126
botocore                     1.29.126
Brotli                       1.0.9
cachetools                   5.2.1
certifi                      2022.12.7
cffi                         1.15.1
cfgv                         3.4.0
charset-normalizer           3.0.1
clearml                      1.10.4
click                        8.1.3
cloudpickle                  2.2.1
cmake                        3.25.0
comm                         0.1.2
contextlib2                  21.6.0
contourpy                    1.0.7
cryptography                 40.0.2
cycler                       0.11.0
Cython                       3.0.2
databricks-cli               0.17.6
dataloader                   2.0
debugpy                      1.6.5
decorator                    5.1.1
defusedxml                   0.7.1
distlib                      0.3.6
Django                       4.1.5
docker                       6.0.1
easydict                     1.10
efficientnet-pytorch         0.7.1
entrypoints                  0.4
et-xmlfile                   1.1.0
executing                    1.2.0
fastapi                      0.89.1
fastjsonschema               2.16.2
filelock                     3.12.0
filterpy                     1.4.5
Flask                        2.3.2
flask-babel                  3.1.0
flatbuffers                  23.5.26
fonttools                    4.38.0
fqdn                         1.5.1
frozenlist                   1.3.3
ftfy                         6.1.1
furl                         2.1.3
future                       0.18.3
gast                         0.4.0
gdown                        4.7.1
gevent                       22.10.2
geventhttpclient             2.0.2
gitdb                        4.0.10
GitPython                    3.1.30
google-api-core              2.11.0
google-auth                  2.16.0
google-auth-oauthlib         1.0.0
google-cloud-core            2.3.2
google-cloud-storage         2.8.0
google-crc32c                1.5.0
google-pasta                 0.2.0
google-resumable-media       2.5.0
googleapis-common-protos     1.59.0
gputils                      1.0.6
greenlet                     2.0.2
grpcio                       1.51.1
gunicorn                     20.1.0
h11                          0.14.0
h5py                         3.9.0
httpcore                     0.17.3
httpx                        0.24.1
huggingface-hub              0.11.1
humanfriendly                10.0
hydra-core                   1.3.1
identify                     2.5.27
idna                         3.4
imageio                      2.24.0
imageio-ffmpeg               0.4.8
imbalanced-learn             0.10.1
imblearn                     0.0
importlib-metadata           6.8.0
imutils                      0.5.4
ipykernel                    6.20.2
ipython                      8.8.0
ipython-genutils             0.2.0
ipywidgets                   8.0.4
isodate                      0.6.1
isoduration                  20.11.0
itsdangerous                 2.1.2
jedi                         0.18.2
jeepney                      0.8.0
Jinja2                       3.1.2
jmespath                     1.0.1
joblib                       1.2.0
jsonpickle                   3.0.1
jsonpointer                  2.3
jsonschema                   4.17.3
jupyter                      1.0.0
jupyter_client               7.4.9
jupyter-console              6.4.4
jupyter_core                 5.1.3
jupyter-events               0.6.3
jupyter_server               2.1.0
jupyter_server_terminals     0.4.4
jupyterlab-pygments          0.2.2
jupyterlab-widgets           3.0.5
kafka-python                 2.0.2
keras                        2.13.1
kiwisolver                   1.4.4
knack                        0.10.1
kornia                       0.7.0
kubernetes                   26.1.0
lap                          0.4.0
lapx                         0.5.4
libclang                     16.0.6
lit                          15.0.7
lmdb                         1.4.1
loguru                       0.7.0
Mako                         1.2.4
Markdown                     3.4.1
MarkupSafe                   2.1.1
matplotlib                   3.6.3
matplotlib-inline            0.1.6
mistune                      2.0.4
mlflow                       2.3.1
mlserver                     1.2.4
mlserver-mlflow              1.3.0
mlxtend                      0.21.0
mpmath                       1.3.0
msal                         1.22.0
msal-extensions              1.0.0
msrest                       0.7.1
msrestazure                  0.6.4
multidict                    6.0.4
munch                        2.5.0
nbclassic                    0.4.8
nbclient                     0.7.2
nbconvert                    7.2.8
nbformat                     5.7.3
ndg-httpsclient              0.5.1
nest-asyncio                 1.5.6
networkx                     3.0
nodeenv                      1.8.0
notebook                     6.5.2
notebook_shim                0.2.2
numpy                        1.23.1
nvidia-cublas-cu11           11.10.3.66
nvidia-cuda-nvrtc-cu11       11.7.99
nvidia-cuda-runtime-cu11     11.7.99
nvidia-cudnn-cu11            8.5.0.96
oauthlib                     3.2.2
omegaconf                    2.3.0
opencv-python                4.5.5.64
openpyxl                     3.1.1
opt-einsum                   3.3.0
orderedmultidict             1.0.1
orjson                       3.8.11
packaging                    23.0
paddle-bfloat                0.1.7
paddlepaddle-gpu             2.5.0
paddleseg                    2.8.0
pandas                       1.5.2
pandocfilters                1.5.0
paramiko                     3.1.0
parso                        0.8.3
pathlib2                     2.3.7.post1
pathspec                     0.11.1
patsy                        0.5.3
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       9.4.0
pip                          23.3.1
pkginfo                      1.9.6
platformdirs                 3.10.0
portalocker                  2.7.0
pre-commit                   3.4.0
pretrainedmodels             0.7.4
prettytable                  3.8.0
prometheus-client            0.15.0
prometheus-flask-exporter    0.22.4
prompt-toolkit               3.0.36
protobuf                     4.23.4
psutil                       5.9.4
ptyprocess                   0.7.0
pure-eval                    0.2.2
py-grpc-prometheus           0.7.0
pyarrow                      11.0.0
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycparser                    2.21
pycryptodome                 3.18.0
pydantic                     1.10.7
Pygments                     2.14.0
PyJWT                        2.4.0
PyNaCl                       1.5.0
pyOpenSSL                    23.1.1
pyparsing                    3.0.9
PyQt5                        5.15.7
PyQt5-Qt5                    5.15.2
PyQt5-sip                    12.11.0
pyrsistent                   0.19.3
pysftp                       0.2.9
PySocks                      1.7.1
python-dateutil              2.8.2
python-dotenv                1.0.0
python-json-logger           2.0.4
python-rapidjson             1.10
pytz                         2022.7.1
PyWavelets                   1.4.1
PyYAML                       6.0
pyzmq                        25.0.0
qtconsole                    5.4.0
QtPy                         2.3.0
qudida                       0.0.4
querystring-parser           1.2.4
rarfile                      4.0
regex                        2023.8.8
requests                     2.28.2
requests-auth-aws-sigv4      0.7
requests-oauthlib            1.3.1
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rsa                          4.9
s3transfer                   0.6.0
scikit-image                 0.19.3
scikit-learn                 1.3.0
scipy                        1.10.0
seaborn                      0.12.2
SecretStorage                3.3.3
segmentation-models-pytorch  0.3.2
Send2Trash                   1.8.0
setuptools                   58.1.0
six                          1.16.0
smmap                        5.0.0
sniffio                      1.3.0
soupsieve                    2.3.2.post1
SQLAlchemy                   2.0.12
sqlparse                     0.4.3
stack-data                   0.6.2
starlette                    0.22.0
starlette-exporter           0.15.1
statsmodels                  0.13.5
sympy                        1.12
tabulate                     0.9.0
tb-nightly                   2.15.0a20230902
tensorboard                  2.13.0
tensorboard-data-server      0.7.1
tensorboard-plugin-wit       1.8.1
tensorboardX                 2.6.2
tensorflow                   2.13.0
tensorflow-estimator         2.13.0
tensorflow-io-gcs-filesystem 0.33.0
termcolor                    2.3.0
terminado                    0.17.1
thop                         0.1.1.post2209072238
threadpoolctl                3.1.0
tifffile                     2022.10.10
timm                         0.6.12
tinycss2                     1.2.1
tomli                        2.0.1
torch                        2.0.0+cu118
torchaudio                   2.0.1+cu118
torchmetrics                 0.10.2
torchsummary                 1.5.1
torchvision                  0.15.1+cu118
tornado                      6.2
tqdm                         4.64.1
traitlets                    5.8.1
triton                       2.0.0
tritonclient                 2.33.0
typing_extensions            4.4.0
ultralytics                  8.0.6
uri-template                 1.2.0
urllib3                      1.26.14
uvicorn                      0.22.0
uvloop                       0.17.0
virtualenv                   20.23.0
visualdl                     2.5.3
wcwidth                      0.2.6
webcolors                    1.12
webencodings                 0.5.1
websocket-client             1.4.2
Werkzeug                     2.3.3
wheel                        0.38.4
widgetsnbextension           4.0.5
wrapt                        1.15.0
yacs                         0.1.8
yapf                         0.40.1
yarl                         1.9.2
zipp                         3.11.0
zope.event                   4.6
zope.interface               6.0

apt list --installed cuda*
Listing... Done
cuda-11-8/unknown,now 11.8.0-1 amd64 [installed]
cuda-cccl-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-command-line-tools-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-compiler-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-cudart-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-cudart-dev-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-cuobjdump-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-cupti-11-8/unknown,now 11.8.87-1 amd64 [installed,automatic]
cuda-cupti-dev-11-8/unknown,now 11.8.87-1 amd64 [installed,automatic]
cuda-cuxxfilt-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-demo-suite-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-documentation-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-driver-dev-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-drivers-545/unknown,now 545.23.06-1 amd64 [installed,automatic]
cuda-drivers/unknown,now 545.23.06-1 amd64 [installed,automatic]
cuda-gdb-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-keyring/unknown,now 1.1-1 all [installed]
cuda-libraries-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-libraries-dev-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-memcheck-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-nsight-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-nsight-compute-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-nsight-systems-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-nvcc-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-nvdisasm-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-nvml-dev-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-nvprof-11-8/unknown,now 11.8.87-1 amd64 [installed,automatic]
cuda-nvprune-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-nvrtc-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-nvrtc-dev-11-8/unknown,now 11.8.89-1 amd64 [installed,automatic]
cuda-nvtx-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-nvvp-11-8/unknown,now 11.8.87-1 amd64 [installed,automatic]
cuda-profiler-api-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-repo-ubuntu2004-11-4-local/now 11.4.4-470.82.01-1 amd64 [installed,local]
cuda-runtime-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-sanitizer-11-8/unknown,now 11.8.86-1 amd64 [installed,automatic]
cuda-toolkit-11-8-config-common/unknown,now 11.8.89-1 all [installed,automatic]
cuda-toolkit-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-toolkit-11-config-common/unknown,now 11.8.89-1 all [installed,automatic]
cuda-toolkit-config-common/unknown,now 12.3.52-1 all [installed,automatic]
cuda-tools-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]
cuda-visual-tools-11-8/unknown,now 11.8.0-1 amd64 [installed,automatic]

could it be the cuda 11.8?

taohan10200 / IIM

segfault with pytorch 1.17 and 2.0.0 #37