vulkaninfo Segmentation fault (core dumped)

[brian2lee]

I was going to try maniskill, following the instruction to install vulkan, while vulkaninfo showed Segmentation fault (core dumped). Followed the troubleshooting, /usr/share/vulkan/icd.d/nvidia_icd.json & /usr/share/glvnd/egl_vendor.d/10_nvidia.json exist. Have no idea what's wrong. Posted same issue on maniskill. Environment:

ubuntu 20.04
nvidia-smi 
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   49C    P0             26W /   80W |      15MiB /   6144MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1318      G   /usr/lib/xorg/Xorg                              4MiB |
+-----------------------------------------------------------------------------------------+

ldconfig -p | grep libGLX_nvidia
    libGLX_nvidia.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libGLX_nvidia.so.0

nvidia_icd.json:

{
    "file_format_version" : "1.0.0",
    "ICD" : {
        "library_path" : "libEGL_nvidia.so.0"
    }
}

10_nvidia.json

{
    "file_format_version" : "1.0.1",
    "ICD": {
        "library_path": "libGLX_nvidia.so.0",
        "api_version" : "1.3.280"
    }
}

And just in case here's my pip3 list:

Package                       Version             
----------------------------- --------------------
absl-py                       2.1.0               
actionlib                     1.14.0              
angles                        1.9.13              
apturl                        0.5.2               
astunparse                    1.6.3               
attrs                         19.3.0              
autobahn                      17.10.1             
Automat                       0.8.0               
base-local-planner            1.17.3              
bcrypt                        3.1.7               
blinker                       1.4                 
bondpy                        1.8.6               
breezy                        3.0.2               
Brlapi                        0.7.0               
cachetools                    5.5.0               
camera-calibration            1.17.0              
camera-calibration-parsers    1.12.0              
catkin                        0.8.10              
catkin-pkg                    1.0.0               
catkin-pkg-modules            1.0.0               
catkin-tools                  0.9.2               
cbor                          1.0.0               
certifi                       2019.11.28          
cffi                          1.17.0              
chardet                       3.0.4               
Click                         7.0                 
cloudpickle                   3.0.0               
colorama                      0.4.3               
command-not-found             0.3                 
configobj                     5.0.6               
constantly                    15.1.0              
controller-manager            0.20.0              
controller-manager-msgs       0.20.0              
cryptography                  2.8                 
cupshelpers                   1.0                 
cv-bridge                     1.16.2              
cycler                        0.10.0              
Cython                        0.29.14             
dacite                        1.8.1               
dbus-python                   1.2.16              
defer                         1.0.6               
defusedxml                    0.6.0               
Deprecated                    1.2.7               
diagnostic-analysis           1.11.0              
diagnostic-common-diagnostics 1.11.0              
diagnostic-updater            1.11.0              
distro                        1.4.0               
distro-info                   0.23ubuntu1         
docutils                      0.16                
dulwich                       0.19.15             
duplicity                     0.8.12.0            
dynamic-reconfigure           1.7.3               
empy                          3.3.2               
entrypoints                   0.3                 
Farama-Notifications          0.0.4               
fast-kinematics               0.2.2               
fasteners                     0.14.1              
fastimport                    0.9.8               
filelock                      3.15.4              
flatbuffers                   24.3.25             
fsspec                        2024.6.1            
future                        0.18.2              
gast                          0.4.0               
gazebo-plugins                2.9.2               
gazebo-ros                    2.9.2               
gencpp                        0.7.0               
geneus                        3.0.0               
genlisp                       0.4.18              
genmsg                        0.6.0               
gennodejs                     2.0.2               
genpy                         0.6.15              
gitdb                         4.0.11              
GitPython                     3.1.43              
glfw                          2.7.0               
google-auth                   2.34.0              
google-auth-oauthlib          1.0.0               
google-pasta                  0.2.0               
gpg                           1.13.1              
grpcio                        1.66.1              
gym                           0.26.2              
gym-notices                   0.0.8               
gymnasium                     0.29.1              
gymnasium-robotics            1.2.4               
h5py                          3.11.0              
httplib2                      0.14.0              
huggingface-hub               0.24.6              
hyperlink                     19.0.0              
idna                          2.8                 
image-geometry                1.16.2              
imageio                       2.35.1              
importlib-metadata            8.4.0               
incremental                   16.10.1             
interactive-markers           1.12.0              
jinja2                        3.1.4               
joint-state-publisher         1.15.1              
joint-state-publisher-gui     1.15.1              
keras                         2.13.1              
keyring                       18.0.1              
kiwisolver                    1.0.1               
language-selector             0.1                 
laser-geometry                1.6.7               
launchpadlib                  1.10.13             
lazr.restfulclient            0.14.2              
lazr.uri                      1.0.3               
libclang                      18.1.1              
lockfile                      0.12.2              
louis                         3.12.0              
lxml                          5.3.0               
lz4                           3.0.2+dfsg          
macaroonbakery                1.3.1               
Mako                          1.1.0               
mani-skill                    3.0.0.dev13         
Markdown                      3.7                 
MarkupSafe                    2.1.5               
matplotlib                    3.1.2               
message-filters               1.16.0              
monotonic                     1.5                 
more-itertools                4.2.0               
mpi4py                        3.0.3               
mplib                         0.2.1               
mujoco                        2.3.7               
netifaces                     0.10.4              
networkx                      3.1                 
nose                          1.3.7               
numpy                         1.24.3              
oauthlib                      3.1.0               
olefile                       0.46                
opencv-python                 4.10.0.84           
opt-einsum                    3.3.0               
osrf-pycommon                 2.0.2               
packaging                     24.1                
paramiko                      2.6.0               
pettingzoo                    1.24.3              
pexpect                       4.6.0               
pillow                        10.4.0              
pip                           20.0.2              
protobuf                      4.25.4              
psutil                        5.5.1               
py-ubjson                     0.14.0              
pyasn1                        0.4.2               
pyasn1-modules                0.2.1               
pycairo                       1.16.2              
pycparser                     2.22                
pycrypto                      2.6.1               
pycryptodomex                 3.6.1               
pycups                        1.9.73              
pydot                         1.4.1               
pygit2                        1.13.3              
PyGithub                      1.43.7              
Pygments                      2.3.1               
PyGObject                     3.36.0              
PyHamcrest                    1.9.0               
PyJWT                         1.7.1               
pymacaroons                   0.13.0              
PyNaCl                        1.3.0               
PyOpenGL                      3.1.0               
pyOpenSSL                     19.0.0              
pyparsing                     2.4.6               
pypng                         0.0.20              
PyQRCode                      1.2.1               
PyQt5                         5.14.1              
pyRFC3339                     1.1                 
pyserial                      3.4                 
python-apt                    2.0.1+ubuntu0.20.4.1
python-dateutil               2.7.3               
python-debian                 0.1.36ubuntu1       
python-gitlab                 2.0.1               
python-gnupg                  0.4.5               
python-qt-binding             0.4.4               
python-snappy                 0.5.3               
PyTrie                        0.2                 
pytz                          2019.3              
pyxdg                         0.26                
PyYAML                        5.3.1               
qt-dotgraph                   0.4.2               
qt-gui                        0.4.2               
qt-gui-cpp                    0.4.2               
qt-gui-py-common              0.4.2               
reportlab                     3.5.34              
requests                      2.22.0              
requests-oauthlib             2.0.0               
requests-unixsocket           0.2.0               
resource-retriever            1.12.7              
roman                         2.0.0               
rosbag                        1.16.0              
rosboost-cfg                  1.15.8              
rosclean                      1.15.8              
roscreate                     1.15.8              
rosdep                        0.25.1              
rosdep-modules                0.25.1              
rosdistro                     0.9.1               
rosdistro-modules             0.9.1               
rosgraph                      1.16.0              
rosinstall                    0.7.8               
rosinstall-generator          0.1.23              
roslaunch                     1.16.0              
roslib                        1.15.8              
roslint                       0.12.0              
roslz4                        1.16.0              
rosmake                       1.15.8              
rosmaster                     1.16.0              
rosmsg                        1.16.0              
rosnode                       1.16.0              
rosparam                      1.16.0              
rospkg                        1.5.1               
rospkg-modules                1.5.1               
rospy                         1.16.0              
rosserial-python              0.9.2               
rosservice                    1.16.0              
rostest                       1.16.0              
rostopic                      1.16.0              
rosunit                       1.15.8              
roswtf                        1.16.0              
rqt-action                    0.4.9               
rqt-bag                       0.5.1               
rqt-bag-plugins               0.5.1               
rqt-console                   0.4.12              
rqt-dep                       0.4.12              
rqt-graph                     0.4.14              
rqt-gui                       0.5.3               
rqt-gui-py                    0.5.3               
rqt-image-view                0.4.17              
rqt-launch                    0.4.9               
rqt-logger-level              0.4.12              
rqt-moveit                    0.5.11              
rqt-msg                       0.4.10              
rqt-nav-view                  0.5.7               
rqt-plot                      0.4.13              
rqt-pose-view                 0.5.11              
rqt-publisher                 0.4.10              
rqt-py-common                 0.5.3               
rqt-py-console                0.4.10              
rqt-reconfigure               0.5.5               
rqt-robot-dashboard           0.5.8               
rqt-robot-monitor             0.5.15              
rqt-robot-steering            0.5.12              
rqt-runtime-monitor           0.5.10              
rqt-rviz                      0.7.0               
rqt-service-caller            0.4.10              
rqt-shell                     0.4.11              
rqt-srv                       0.4.9               
rqt-tf-tree                   0.6.4               
rqt-top                       0.4.10              
rqt-topic                     0.4.13              
rqt-web                       0.4.10              
rsa                           4.9                 
rtabmap-python                0.21.5              
Rtree                         1.3.0               
ruamel.yaml                   0.18.6              
ruamel.yaml.clib              0.2.8               
rviz                          1.14.25             
sapien                        3.0.0.dev2          
scipy                         1.10.1              
screen-resolution-extra       0.0.0               
SecretStorage                 2.3.1               
sensor-msgs                   1.13.1              
service-identity              18.1.0              
setuptools                    45.2.0              
simplejson                    3.16.0              
sip                           4.19.21             
six                           1.14.0              
smach                         2.5.2               
smach-ros                     2.5.2               
smclib                        1.8.6               
smmap                         5.0.1               
squaternion                   2023.9.2            
systemd-python                234                 
tabulate                      0.9.0               
tensorboard                   2.13.0              
tensorboard-data-server       0.7.2               
tensorflow                    2.13.1              
tensorflow-estimator          2.13.0              
tensorflow-io-gcs-filesystem  0.34.0              
termcolor                     2.4.0               
tf                            1.13.2              
tf-conversions                1.13.2              
tf2-geometry-msgs             0.7.7               
tf2-kdl                       0.7.7               
tf2-py                        0.7.7               
tf2-ros                       0.7.7               
topic-tools                   1.16.0              
toppra                        0.6.0               
torch                         1.8.1+cu101         
torchaudio                    0.8.1               
torchvision                   0.9.1+cu101         
tqdm                          4.66.5              
transforms3d                  0.4.2               
trimesh                       4.4.8               
turtlebot3-teleop             1.2.5               
Twisted                       18.9.0              
txaio                         2.10.0              
typing-extensions             4.5.0               
u-msgpack-python              2.1                 
ubuntu-advantage-tools        8001                
ubuntu-drivers-common         0.0.0               
ufw                           0.36                
unattended-upgrades           0.1                 
urllib3                       1.25.8              
usb-creator                   0.3.7               
vcstools                      0.1.42              
wadllib                       1.3.3               
werkzeug                      3.0.4               
wheel                         0.34.2              
wrapt                         1.11.2              
wsaccel                       0.6.2               
wstool                        0.1.18              
xacro                         1.14.18             
xkit                          0.0.0               
zipp                          1.0.0               
zope.interface                4.7.1  

[charles-lunarg]

I believe this is actually a bug during shutdown of Nvidia's Vulkan Safety critical driver. If I install the nvidia-driver-560 and run vulkaninfo, I get the output I expect followed by segmentation fault ( core dumped). What makes me think its the driver's fault is that I get the same output from vkcube and the crash occurs after vkDestroyInstance successfully returned (I did a debug build to confirm this).

This may be the same issue as #1025 as both have nvidia driver 560.

[brian2lee]

@charles-lunarg yeah somehow I reinstall my driver to nvidia 470 it works, at least for vulkaninfo. Just got into this thing, got to wait and see if any furthur issues appears.

[StoneT2000]

@charles-lunarg thanks for helping debug this issue. We had a few other users of our vulkan based rendering software have the same issue and couldn't figure it out so I pointed them here. Is there a reliable way to debug driver issues with seg faults? E.g. if there is a new nvidia driver, should we always recommend people to downgrade the nvidia driver?

[charles-lunarg]

Is there a reliable way to debug driver issues with seg faults?

It is much easier when the fault occurs inside a driver rather than after main returns. Because then you'd have a stack trace pointing to something.

I personally believe this is just the teething problems of Vulkan Safety critical being shipped for the first time - the driver is installed but is erroneously being used by the regular Vulkan-Loader. There may be a way for the loader to detect this situation and prevent loading of the safety critical driver, but since its already shipped thats a moot point as fixing the driver is the best action. I have gone ahead and contacted the Nvidia developer I know who works on drivers.

So for the time being I would recommend downgrading.